Tales from Barcelona - Day2

Day 2

Some Notes from TheServerSide Java Symposium Europe - day 2

Drunken Chats

Not really too drunk last night, but spent a fair chunk of the evening chatting with Oisin Hurley from Iona. Oisin (pronounced like Ocean)

had earlier been on the panel as an "SOA expert", and knows a thing or two about architecting and delivering software projects.

One thing we talked about project life-cycles and how there is a gap in the market for a company/organisation to provide a very prescriptive way of setting up a project - based on the project, some templated project that prescribes how the team will develop. For example - if someone came in and said: "Right, here's your environment. Its: Maven, Subversion, CruiseControl, Eclipse, TomCat, Oracle. Secondly, Here's your application architecture for this project: JSF/myFaces, Hibernate, no/few singletons, messaging through Mule, other coding standards, JUnit / MockRunner - 60% minimum coverage."

That takes away so many problems from the development team - the framework in which you code is prescribed, so you can just get on with adding business value. The "lucky" thing for .Net developers is that they have fewer implementation choices, which actually concentrates the mind on writing the code to solve the business case.

Enterprise Application Mashup: Architecting for the Future

The speaker here was Eugene Ciurana - a pretty charismatic guy with good references. Previously he worked for several years architecting the systems for the worlds biggest retailer: Walmart.

His basic message was that we (the development industry) should move further away from custom coding and more into "Smart" coding and Integration.

This message sounds very sensible of course. He also warned about some common mistakes he has seen in enterprises:

• blindly buying, rather than building in some cases - this is where the word "Smart" comes in.
• Outsourcing key projects to third-parties (as the in-house knowledge is high-value)
• Cost overruns come from some common sources too:
  • Remote management is expensive
  • Deployment & Integration costs are often more expensive than proposed

He also confirmed the industry notion that 40% of development is spent maintaining code (i.e. not adding any business value), although this figure is higher in some companies.

One of the things that he managed to achieve was to persuade the techs, business people & lawyers at Walmart to use Open Source technologies - apparently the lawyers were the hardest - as they want indemnity.

He felt that we should embrace Open Source - and always use recent versions and cutting edge stuff. Now this is seen as a high risk from enterprises - but his experience was that it worked. I think his rule of thumb was that you should never run software more than 18 months past its General Release date. He contended that this rule had resulted in the most stable software. As soon as people went against this rule and stayed with the "safe" versions - that's when maintenance costs actually started to go up and availability of the application fell.

Resin upgrade anyone?..

The second half of his slot focused on integration & middleware: "Get rid of point-to-point communication" was basically his message. The way he suggests is using an ESB (Enterprise Service Bus). I totally agree. He used Mule - which has had various independent positive reviews and is in production in many high demand environments. Interestingly, they even routed their CMS through the Bus, such that content was distributed out to servers through the ESB also. This design served him well, as they ended up having to replace the CMS system, but because they were going through the Bus, there was zero impact on other parts of the system (as the BUS enforces a common format & API contract)

One other thing he recommended was that we but Java 6 or Java 7 on our technology roadmap NOW.

Again I agree - I think we should go up to Java 6 soon and get specific training to get our developers up to speed on the new features of Java compared to 1.4

One of the key benefits to the upgrade from 1.4 will be the improved runtime monitorability (is that even a word?!) of the JVM through JMX.

Case Study: High Volume Processing using Spring Framework

The speaker was Meeraj Kunnumpurath (Enterprise Architect @ Voca)

This title was actually a bit misleading, as the high volume processing was not achieved due to their usage of Spring, but rather other technology choices they had made. His response to that claim: "Spring encourages best practices, which fit in well with an architecture that provides performance". Still a very interesting talk. Here are some key points:

• Facts & Figures - Voca is pretty hard-core!
  • Voca processes all UK direct debits, 90% of UK salary payments, and a bunch of other stuff
  • This equates to 100 million transactions daily
  • In testing, their architecture has processed 100 million transactions in 2 hours 15 mins. That's about 12,000 per second. Not bad.
  • The process is considered part of the UK Critical National Infrastructure - alongside things like water & electricity distribution - due to the effect that its failure would have on the economy
  • The total value of transactions they process yearly is €4.5 trillion - yes, trillion
• Design
  • Technology Stack: Java, Spring, Hibernate, WebLogic, Oracle DB - 10g RAC, Oracle AQ (Advanced Queuing), , Soalaris Sun Fire UltraSPARC
  • Whilst they used Hibernate for most things - the very core parts of the system, they dropped back down to JDBC calls.
    It seemed this was just to be "safe" as Hibernate was not considered proven enough when they started the project.
    He did not indicate whether they would make that decision differently now...
  • All transactions were local - not XA (i.e. distributed)
  • The choice of Oracle AQ as the messaging was an interesting one.
    Oracle AQ implements its queueing using tables, and one of the interfaces it exposes is through JDBC, rather than JMS.
    This means:
    • No additional messaging server to worry about
    • Writes into the messaging queues are part of the same transaction as any other DB writes - cool! Nice and neat way of avoiding XA
  • Minimise Movement of Data
    If they had logic that required a fair bit of data, they moved that processing down into the db (as stored procs presumably)
  • Parallelisation
    Break up any processing, and do the bits concurrently where possible
  • Partition Data
    Similar to minimising movement of data, if you are able partition data this provides performance boosts.
  • Minimising DB contention
    They were using RAC, and by pinning app servers to a single node in the RAC this greatly reduced risk of contention at DB level
  • Asynchronous decoupling
  • Caching
    • Hibernate 2nd level cache
    • Plus some of theur own custom caching
  • Use of Spring allowed:
    • Developers didn't have to run in an container at all - just straight out of SE with some db pool, transaction mgmnt, etc. injected in
      Just a config change to go to live versions
    • promoted loose coupling & modularity
    • increased productivity by 30% (over using the full EE stack)
  • They relied "hugely" on Oracle for:
    • bulk uploads
    • High Availability - they used Data Guard (I don't know much about that)
    • Load Balancing
  • They also used Mule (ESB) for service provisioning and orchestration
  • All requests were stateless. State was maintained in the DB
• Other bits & pieces
  • They found it difficult to sell the use of Open Source (Spring, Hibernate) into the ultra-conservative UK banking sector - but they did it

The thing that Meeraj said to watch for the future is SCA (Service Component Architecture) - a bit buzzwordy, but something to keep an eye on...

Summary: These guys are at the front end of high performance java apps, and they aren't using any of the fancy grid technology stuff out there (like GigaSpaces & Appistry). Just goes to show that with the right architecture and with performance in mind from day 1 you can go a long way... Also interesting to hear of yet another Case Study that relied significantly on the features provided by Oracle.

Eclipse Persistence Services Project

By Doug Clark (who did yesterday afternoon's KeyNote that I mentioned on yesterday's blog)

This was a drill down into the one bit of that - the new incarnation of TopLink: EclipseLink

Firstly, I don't like the name! It implies that it is some kind of Eclipse plugin or something - and not a full persistence framework

Anyway, some more notes:

• JPA1.0 compliant
• Also offers:
  • MOXy - For Object to XML mapping
    JAX-B comliant
    Developer can treat an xml doc as an object
    Mapping (columns to fields) can be auto, or controlled through meta data
  • SDO - Service Data Objects
  • DBWS - Database Web Services
    This was pretty interesting - basically allows you to expose an SQL statement as a Web Service.
    e.g. SELECT * FROM S_LANGUAGES - we could expose that as a GetLanguages Web Service with very little effort.
  • EIS - (Enterprise Integration <Something> - I think) This allows you to expose non database data sources - such as XML or JMS and access them in the same way that you would a db
• Runs in any java container (doesn't have to be EE)
• Entity caching
  Interestingly it caches the actual object instance rather than the pure data representation. This has performance benefits, and is unique in the ORM space I believe
• Level 2 caching across transactions & users
• Coordination in a clustered environment
  Can be (configurably) through JMS, RMI, JGroups, etc.
  Also integrates with Coherence (formerly Tangosol) which is the coolest distributed caching around...
• Default is optimistic caching
• Caching options are configurable
  e.g. expiry, pessimistic, etc.
• Doug hammered home the importance of data locking with regards to concurrency.
  There are defaults, but you have to think about how you want your app to work.
  this can be optimistic locking, which checks on a per column level when updating - and there are other options
• Supports JPQL as well as SQL
• Optimisation - e.g. for joins & batches
• Annotations allow interesting configurations. Couple of interesting examples:
  • @PrivateOwned - if an attribute is annotated like this, then should you remove the refernce from the holding object, then it is also physically deleted from the store. e.g. customer & address - if you delete the customer, you want to auto-delete the address (not just the reference)
  • @Converters - defines how the annotated attribute should be converted on its way to a DB type (and in other direction)

• Customisable - for example you can plug in factories to define object creation, or object copy behavaiour
• Performance - Doug claimed it is the most optimised persistence framework out there - and I believe that.

Doug didn't mention any other O/R mapping tools during the presentation, but inevitably the questions were asked about comparisons to Hibernate.

Main thing is that Hibernate provides the ORM stuff, but not the other features listed above.

Also, Doug claims that it is faster - which I believe, but would need to be verified of course :-)

Summary: EclipseLink is a really robust, really tunable persistence framework that I think we should take a serious look at - especially now that the whole framework is open source.

GlassFish Update

For those that don't know, GlassFish is an Open Source, Enterprise Class Java EE application server (and a bunch more)

Sun is managing the project and probably the most surprising thing for me is the traction that its gained in a relatively short space of time.

Anyway, there are plenty of real-world implementation out there already - mainly on v1. v2 is just released and it has a lot of cool stuff.

Here are a few highlights that I jotted down as interesting:

• Memory Replication (using JXTA) - for session sharing as well as caching
• Built in ESB with JBI - with GUI admin (or CLI)
• Nice adminsitration UI
• JPA implementation is Toplink
• The JMS implementation is actually very good - enterpise class

Summary: GlassFish is a definite contender as application server choice should we ever replace ours

OSGi

Its all about component integration, where components can be anything - from mobile devices to software libraries.

Effectively provides a gluing protocol between the components, and includes support for dependencies & versioning.

Key point is that its all dynamic (can be managed at runtime - new versions deployed, things brought out of service - all the dependents automatically become out of service too, etc...)

All us developers at VC have been using it for years - its the underpinning of Eclipse's plugin framework :-)

Can't be bothered writing more so here's a link to the presentation if you're interested.

High Performance Architectures

There was also an interesting session on High Performance Architectures which had dudes from GigaSpaces, Azul, Appistry & Kirk Pepperdine on the panel. Was interesting, but if I don't stop writing now, I won't have any drunk chat to add tomorrow... Ask me when I'm back in GIB...

Tales from Barcelona

Some Notes from TheServerSide Java Symposium Europe

Drunken Chats

Arrived yesterday and spent the evening eating and drinking with some of the speakers.

Interestingly, the conversation turned to gambling for a while with some heated debate about the Monty Hall Problem and some variations of it.

Its reassuring to see that even Kirk Pepperdine who has a degree in Maths had to think pretty hard about this stuff.

Gil Tene came up with the following situation:

In my two closed hands I have two amounts: x and 2x - you don't know the value of x, so the amounts could be €5 & €10, or €10 & €20, etc.

You pick one hand and I show you the contents. Now: is there any advantage to switching hands at this point?

Instinctively we would say there is no advantage (and I am sure there isn't) but Kirk's argument was that you should always switch based on the perceived value:

i.e. there are two outcomes when you switch - you will either double your money (as the other hand has 2x) or you will half your money (as your original pick was 2x, so the other hand must contain x)

Based on that thinking you should always switch, as you might double it, and the worst that can happen is that you will lose half - that's good value.

However the rest of us at the table contended that it was entirely arbitrary and a player that always switched would do no better (or worse) than a player that never switched. Anyway I promised to knock up a simulation and run it a few million times to get a definitive answer...

I talked with Gil about performance monitoring and we discussed T2000s boxes that we use as servers. He made an interesting observation: the OS reports CPU usage based on the number of idle cycles per possible thread. Now on the T2000 has 8 CPU cores - each able to run 4 threads "concurrently". Now the way that this "concurrency" works is that if one thread is held up waiting for a memory read, IO or lock or something, then it will swap in another thread in the mean time, maximising the usage of the the core. The "concurrency" therefore depends on the thread having to wait for non-CPU resources. Now here's the interesting issue: If you have a CPU intensive (but not using memory) process (say: something that's just doing some intensive integer number crunching) each thread is just in the CPU - so there is no "down time" for other threads to get in - so each core is only really processing one thread at a time. This is fine, after all, you can't change the laws of physics - the problem is that when you ask the OS about how much CPU resource is being used (using top) because it is reporting on processor time per thread it can be drastically under-represented. If we kick off 32 threads all running pure CPU-time processes then apparently the OS will never report more than 25% usage.

Gil had seen companies that had there systems to alert them whenever CPU went over, say 60% - but they never got alerted - even when the machine was actually max'ed out!

Now fortunately our resin threads do not fall under the category of pure CPU-time activity, so we would certainly see our CPU go over 25% if we were heavily used. Nevertheless - this is something that IDG should keep an eye on, as the OS may be reporting not-maxed-out CPU activity despite the fact that we are actually max'ed out in terms of processing...

Day 1

Key-Note: Language Oriented Programming

This morning was the Key Note speech from Martin Fowler concerning LOP (Language Oriented Programming). Was interesting to listen to and basically concerned the fact that java as a language does not do a great job with readability. DSLs (Domain Specific Languages) was a buzz topic last year, and this is along the same lines: There are advantages to be gained from placing an abstraction layer on top of the underlying language: it makes it easier to read (even by "the business") and removes unnecessary code providing a stronger context. Pretty difficult to describe without examples, so I'll attach the presentation notes when they become available...

"Case Study: The New Guardian.co.uk"

After that I attended "Case Study: The New Guardian.co.uk" co-presented by Matt Wall (Tech Architect @ guardian) and Erik Doernenburg (TA @ ThoughtWorks - consultants that implemented it)

I found this very interesting as they basically just replaced their systems and re-did their Content Management. Sound familiar?!

Guess what their new technology stack is?

Apache, Resin, Oracle - also sound familiar?!

There site obviously has a very different model to ours - their focus is high volume content serving, whilst ours is high transaction throughput.

Nevertheless, some interesting points which I have noted down below:

• Facts & Figures
• They have about 2 million articles
• They serve up to 500 page requests per second - averaging at 150 million per month
• They have "hundreds" of apache servers
• They currently have 4 resin servers
• Each resin server runs at around 1000 threads - and can peak up to 2000 threads (during failover for example)
• They are running on Dell hardware
• When designing the architecture here are some decisions they made:
• Not to use full Java EE stack
  They did not want the "weight" of a full EE system (i.e. with EJBs, etc)
  They are just deploying WARs, not EARs
• Not to use JSP - they chose velocity as their templating system
  This is because they wanted to take complexity away from the web page designers (who were page designers, not developers)
  Also - they wanted to take away the power from web designers that JSP pages offer - i.e. a restricted & controlled functionality set.
• They used the Yahoo Web Toolkit (YUI ) for the backoffice front-ends.
  They may have chosen differently today, but they started this 2 years ago - and YUI is a very decent, rich & mature web component library - so maybe they would have chosen the same one...
• All requests are stateless
  Obviously, not having to worry about state greatly facilitates scalability
• They used SpringMVC as their framework, and seemed very happy with it.
  It provided the request level transactions for example.
• They used Hibernate as their O/R mapping tool
• Caching - They used EHCache at two levels:
• Plugged into hibernate to provide data-level caching
• At the front end, to cache the rendered HTML generated by the components.
• Methodology
• They were strictly model driven.
  The business owned the business model which was shared between the Business and developers.
  The implementation model was derived from that.
  Developers were not able to change the business model. This worked well as it provided:
• Common vocabulary between Business & IT
• All significant relationships are shown on the model & therefore known by Business & IT
• The model itself took on a certain "momentum" which meant that there were not frequent, significant changes to the model
• The model wen all the way through to the velocity templates, so that even the web designers were working off the same model.
• They ensured that all project members (PM, BAs, devs (internal & consultants), testers, etc.) were all in the same room.
  They said that had been a great thing.
• They had stuck to pair programming - principally as a way of sharing knowledge through the team.
• Continuous performance tests (as well as unit tests obviously)
  Performance was key, so they ran continuous (nightly I think) benchmarks to see if any code had degraded performance.
  They set certain targets on their test environment which they knew would extrapolate upwards to the live environment.
  I think we should be doing something similar - as we only worry about performance when it starts biting us in the ass.
• They have jProfiler running always (presumably only in test env) which took the guesswork out of performance bottlenecks
• Other interesting bits & bobs:
• They supported draft content - which did get store in the DB - and the work-flow (approval) was managed through the DB
• They indicated that the "Materialised Views" feature in Oracle 10g had made massive improvements to their performance also. I had not heard of this previously, but apparently its about optimising join queries by pre-indexing the join data...
• Why didn't they choose an off-the-shelf CMS product?
  Apparently they wanted to represent their model their way, and not be forced into a vendor-conceived notion of what their content model should look like. I wanted to follow up on this, but ran out of time... I will grab him over the next couple of days...
• The final question I asked: "How much has the project cost?"
  He smiled ruefully and finally said "Lets just say several million" - apparently mainly in consultancy fees & hardware

Summary: their experience shows that you can certainly achieve enterprise level performance using open source products. I liked many aspects of their development methodology, but wonder if they were to start again whether they would work with a CMS vendor...

Afternoon KeyNote: Oracle "going to the extreme"

Doug Clark (Oracle Principal Product Manager) talked about where Oracle is at with their application stack at the moment.

Here are the highlights of what's happened in last 12 months:

• Oracle's well respected O/R mapping tool TopLink has been released to the community as open source under the Eclipse license.
• It is now called EclipseLink
• This is supposed to be a really good O/R mapping product (as you'd expect from Oracle) and now that it is free to use I see it gaining a lot more traction
• It is integrated with Eclipse and they showed a brief demo - basically it connects up to the target db and facilitates the mapping of db tables & columns to the relevant pojos - this mapping is stored as annotations, but can also be stored as config. Looked pretty good.
• Oracle's JSF component library has also been open sourced & released - in this case it has been integrated into the apache myFaces project.
• Also, Oracle's new Ajax components have been donated over to the apache project.
• They demo'd the JBuilder plugin which allows drag-and-drop style page building using the available JSF components.
  This worked really nicely (as you'd expect) and also allowed some cool stuff like: if you dragged on a chart (based on data mapped through EclipseLink) then you could change a property to determine whether it got generated as flash, jpg, png, etc. Pretty cool looking.
• Oracle SOA Suite
  Suite of SOA (Service Oriented Architecture) supporting products like ESB (Enterprise Service Bus) with the usual features (governance, transformation, orchestration, messaging, etc.)
  Looks strong.
• Grid Computing (scalability)
  Oracle acquired Tangosol a few months back who were the leaders in the grid computing space with their Coherence product. I remember from last year, Cameron Purdy describing how Coherence working was once of the highlights for me - very smart guy, very good product.
  Anyway - now Oracle have it and have incorporated it into their Grid Computing story - which is now an excellent story...
  Apparently its easy to use the grid computing from a coding perspective, so I'm going to get Doug to give me a hands on demo some time in the next couple of days...

Summary: To be honest, I think the Oracle suite of products is really hot right now - and in my opinion are really leading the way in Enterprise Java application development. They gave out DVDs with all the above on, so hopefully I'll get a chance to play some time...