Tales from Barcelona

Some Notes from TheServerSide Java Symposium Europe

Drunken Chats

Arrived yesterday and spent the evening eating and drinking with some of the speakers.

Interestingly, the conversation turned to gambling for a while with some heated debate about the Monty Hall Problem and some variations of it.

Its reassuring to see that even Kirk Pepperdine who has a degree in Maths had to think pretty hard about this stuff.

Gil Tene came up with the following situation:

In my two closed hands I have two amounts: x and 2x - you don't know the value of x, so the amounts could be €5 & €10, or €10 & €20, etc.

You pick one hand and I show you the contents. Now: is there any advantage to switching hands at this point?

Instinctively we would say there is no advantage (and I am sure there isn't) but Kirk's argument was that you should always switch based on the perceived value:

i.e. there are two outcomes when you switch - you will either double your money (as the other hand has 2x) or you will half your money (as your original pick was 2x, so the other hand must contain x)

Based on that thinking you should always switch, as you might double it, and the worst that can happen is that you will lose half - that's good value.

However the rest of us at the table contended that it was entirely arbitrary and a player that always switched would do no better (or worse) than a player that never switched. Anyway I promised to knock up a simulation and run it a few million times to get a definitive answer...

I talked with Gil about performance monitoring and we discussed T2000s boxes that we use as servers. He made an interesting observation: the OS reports CPU usage based on the number of idle cycles per possible thread. Now on the T2000 has 8 CPU cores - each able to run 4 threads "concurrently". Now the way that this "concurrency" works is that if one thread is held up waiting for a memory read, IO or lock or something, then it will swap in another thread in the mean time, maximising the usage of the the core. The "concurrency" therefore depends on the thread having to wait for non-CPU resources. Now here's the interesting issue: If you have a CPU intensive (but not using memory) process (say: something that's just doing some intensive integer number crunching) each thread is just in the CPU - so there is no "down time" for other threads to get in - so each core is only really processing one thread at a time. This is fine, after all, you can't change the laws of physics - the problem is that when you ask the OS about how much CPU resource is being used (using top) because it is reporting on processor time per thread it can be drastically under-represented. If we kick off 32 threads all running pure CPU-time processes then apparently the OS will never report more than 25% usage.

Gil had seen companies that had there systems to alert them whenever CPU went over, say 60% - but they never got alerted - even when the machine was actually max'ed out!

Now fortunately our resin threads do not fall under the category of pure CPU-time activity, so we would certainly see our CPU go over 25% if we were heavily used. Nevertheless - this is something that IDG should keep an eye on, as the OS may be reporting not-maxed-out CPU activity despite the fact that we are actually max'ed out in terms of processing...

Day 1

Key-Note: Language Oriented Programming

This morning was the Key Note speech from Martin Fowler concerning LOP (Language Oriented Programming). Was interesting to listen to and basically concerned the fact that java as a language does not do a great job with readability. DSLs (Domain Specific Languages) was a buzz topic last year, and this is along the same lines: There are advantages to be gained from placing an abstraction layer on top of the underlying language: it makes it easier to read (even by "the business") and removes unnecessary code providing a stronger context. Pretty difficult to describe without examples, so I'll attach the presentation notes when they become available...

"Case Study: The New Guardian.co.uk"

After that I attended "Case Study: The New Guardian.co.uk" co-presented by Matt Wall (Tech Architect @ guardian) and Erik Doernenburg (TA @ ThoughtWorks - consultants that implemented it)

I found this very interesting as they basically just replaced their systems and re-did their Content Management. Sound familiar?!

Guess what their new technology stack is?

Apache, Resin, Oracle - also sound familiar?!

There site obviously has a very different model to ours - their focus is high volume content serving, whilst ours is high transaction throughput.

Nevertheless, some interesting points which I have noted down below:

• Facts & Figures
• They have about 2 million articles
• They serve up to 500 page requests per second - averaging at 150 million per month
• They have "hundreds" of apache servers
• They currently have 4 resin servers
• Each resin server runs at around 1000 threads - and can peak up to 2000 threads (during failover for example)
• They are running on Dell hardware
• When designing the architecture here are some decisions they made:
• Not to use full Java EE stack
  They did not want the "weight" of a full EE system (i.e. with EJBs, etc)
  They are just deploying WARs, not EARs
• Not to use JSP - they chose velocity as their templating system
  This is because they wanted to take complexity away from the web page designers (who were page designers, not developers)
  Also - they wanted to take away the power from web designers that JSP pages offer - i.e. a restricted & controlled functionality set.
• They used the Yahoo Web Toolkit (YUI ) for the backoffice front-ends.
  They may have chosen differently today, but they started this 2 years ago - and YUI is a very decent, rich & mature web component library - so maybe they would have chosen the same one...
• All requests are stateless
  Obviously, not having to worry about state greatly facilitates scalability
• They used SpringMVC as their framework, and seemed very happy with it.
  It provided the request level transactions for example.
• They used Hibernate as their O/R mapping tool
• Caching - They used EHCache at two levels:
• Plugged into hibernate to provide data-level caching
• At the front end, to cache the rendered HTML generated by the components.
• Methodology
• They were strictly model driven.
  The business owned the business model which was shared between the Business and developers.
  The implementation model was derived from that.
  Developers were not able to change the business model. This worked well as it provided:
• Common vocabulary between Business & IT
• All significant relationships are shown on the model & therefore known by Business & IT
• The model itself took on a certain "momentum" which meant that there were not frequent, significant changes to the model
• The model wen all the way through to the velocity templates, so that even the web designers were working off the same model.
• They ensured that all project members (PM, BAs, devs (internal & consultants), testers, etc.) were all in the same room.
  They said that had been a great thing.
• They had stuck to pair programming - principally as a way of sharing knowledge through the team.
• Continuous performance tests (as well as unit tests obviously)
  Performance was key, so they ran continuous (nightly I think) benchmarks to see if any code had degraded performance.
  They set certain targets on their test environment which they knew would extrapolate upwards to the live environment.
  I think we should be doing something similar - as we only worry about performance when it starts biting us in the ass.
• They have jProfiler running always (presumably only in test env) which took the guesswork out of performance bottlenecks
• Other interesting bits & bobs:
• They supported draft content - which did get store in the DB - and the work-flow (approval) was managed through the DB
• They indicated that the "Materialised Views" feature in Oracle 10g had made massive improvements to their performance also. I had not heard of this previously, but apparently its about optimising join queries by pre-indexing the join data...
• Why didn't they choose an off-the-shelf CMS product?
  Apparently they wanted to represent their model their way, and not be forced into a vendor-conceived notion of what their content model should look like. I wanted to follow up on this, but ran out of time... I will grab him over the next couple of days...
• The final question I asked: "How much has the project cost?"
  He smiled ruefully and finally said "Lets just say several million" - apparently mainly in consultancy fees & hardware

Summary: their experience shows that you can certainly achieve enterprise level performance using open source products. I liked many aspects of their development methodology, but wonder if they were to start again whether they would work with a CMS vendor...

Afternoon KeyNote: Oracle "going to the extreme"

Doug Clark (Oracle Principal Product Manager) talked about where Oracle is at with their application stack at the moment.

Here are the highlights of what's happened in last 12 months:

• Oracle's well respected O/R mapping tool TopLink has been released to the community as open source under the Eclipse license.
• It is now called EclipseLink
• This is supposed to be a really good O/R mapping product (as you'd expect from Oracle) and now that it is free to use I see it gaining a lot more traction
• It is integrated with Eclipse and they showed a brief demo - basically it connects up to the target db and facilitates the mapping of db tables & columns to the relevant pojos - this mapping is stored as annotations, but can also be stored as config. Looked pretty good.
• Oracle's JSF component library has also been open sourced & released - in this case it has been integrated into the apache myFaces project.
• Also, Oracle's new Ajax components have been donated over to the apache project.
• They demo'd the JBuilder plugin which allows drag-and-drop style page building using the available JSF components.
  This worked really nicely (as you'd expect) and also allowed some cool stuff like: if you dragged on a chart (based on data mapped through EclipseLink) then you could change a property to determine whether it got generated as flash, jpg, png, etc. Pretty cool looking.
• Oracle SOA Suite
  Suite of SOA (Service Oriented Architecture) supporting products like ESB (Enterprise Service Bus) with the usual features (governance, transformation, orchestration, messaging, etc.)
  Looks strong.
• Grid Computing (scalability)
  Oracle acquired Tangosol a few months back who were the leaders in the grid computing space with their Coherence product. I remember from last year, Cameron Purdy describing how Coherence working was once of the highlights for me - very smart guy, very good product.
  Anyway - now Oracle have it and have incorporated it into their Grid Computing story - which is now an excellent story...
  Apparently its easy to use the grid computing from a coding perspective, so I'm going to get Doug to give me a hands on demo some time in the next couple of days...

Summary: To be honest, I think the Oracle suite of products is really hot right now - and in my opinion are really leading the way in Enterprise Java application development. They gave out DVDs with all the above on, so hopefully I'll get a chance to play some time...