Getting away from the line-by-line details of code is necessary – the big picture matters. Getting away to Berlin, Germany, is a treat. I’ve spent most of a week at the Berlin Buzzwords Conference http://berlinbuzzwords.de listening and talking about data management. It’s hard not to feel spoiled. Kudos to the event organizers; they created a nice environment for like-minded technologists to gather, interact and learn.
Below are a few brief insights and comments from the conference.
Web stacks. Websites are linked by dynamic services – not simple URLs. Sites integrate twitter, facebook, distributed authentication, gravatar, google analytics, URL shortners, advertising networks, game networks… This trend is appearing in all web properties, regardless of size.
The integration of these third-parties has transformed the LAMP stack into a SOA middleware-like stack. User inputs are processed asynchronously by a collection of loosely connected services. Having lived a little in both worlds, it is entertaining to watch the staid world of service oriented architecture middleware meet the hipper web. Ruby on Websphere, anyone?
Data pipelines. People are talking more about data processing pipelines and less about specific data collection and storage tools. Correctly, the processing – and the data flow through a processing pipeline – is coming in to focus over specific storage tools. A simple example – people aren’t asking, as loudly, “how do I collect my webserver logs and where do I store them once collected?” Tools have been built to solve that problem. They’re now asking, “how do I process these logs and extract value from this wealth of data in a timely enough fashion to drive revenue and other forms of business value.”
“How do I ingest and organize firehose sources in real time? What’s the right data flow through collection, real time analytics, long term analytics, long term storage? How do I push the results of analytics into queryable systems so that my services can read and act on my calculated results immediately?”
A wealth of data management technology has been created in the last 5 years. We’re maturing into conversations about integration.
A VoltDB workshop. From code to 100,000 feet to a hands on workshop. Conference proper concluded, I had the pleasure to spend a morning with 12 folks who signed up for a hands-on VoltDB workshop. Conference attendees from England, Spain and Germany participated. Pretty amazing, when you think about it.
We talked in some detail about VoltDB architecture, downloaded the open source product and walked through the components of the VoltDB distribution. After that, we spent about 30 minutes describing and showing the components of a VoltDB application, and explained in piece-by-piece one of our sample applications.
I hope it was useful – I think everyone learned something (I certainly did). We look forward to organizing more meet-ups and workshops soon.
Sr. Software Engineer