Agile Analytics and the SAP Information Design Tool – 1. Collaborative Development

In my previous blog post introducing this series of posts, I discussed both the concept of Agile Analytics (with particular reference to the recent, excellent book by Ken Collier, Agile Analytics) and how the new SAP Information Design Tool (IDT) could go some way (but, by no means, all) to support it.  This blog explores a specific area of detail here, namely how the  IDT facilitates a degree of collaborative development and version control in SAP Universe development.  It’s far from being a complete version control solution – lots of room for improvement – but it’s certainly an advance on what is possible in the old Universe Designer traditionally used by SAP developers and does allow developers to work together on aspects of the single Universe artefact.

In his Agile Analytics book, Collier devotes his eighth chapter to the topic of “Version Control for Data Warehousing”.  Clearly the IDT is not a  Data Warehouse development tool but it certainly qualifies as one of the artefacts used in the delivery of Analytics solutions that Collier lists.  The IDT’s purpose is to prepare and manage the semantic layer on top of Data Warehouses, Marts, etc… rather than actually design one but I think the core principles of that chapter are still applicable. 

Collier writes that ‘Version control is mandatory on all Agile Analytics projects’ and provides five advantages to the delivery process. 
1) It allows development rewind. 
2) It allows controlled sharing of code. 
3) It maintains an audit trail of development. 
4) It provides release control.
5) It encourages ‘Fearlessness’ in development. 

I particularly like that word ‘Fearlessness’ and the understanding that it allows developers to experiment with solutions without adversely affecting the project.  This, as part of a collaborative development environment, is perhaps where the IDT can offer the most support over the traditional Universe Designer – always a nightmare-ish tool to promote code through – although, again, it’s far from complete in supporting all five of Collier’s version control advantages i.e. it doesn’t really support the rewind principle beyond one iteration or maintain an audit trail of development. 

Let’s focus instead on what the IDT does deliver.  I’ll cover how different developers can work on the same SAP Universe, applying a degree of version control to and sharing the code of, the different layers of the ‘exploded’ Universe.  Throughout these developers will have to work within the principles of good communication and collaborative working  (because the IDT won’t do it all – no tool would!). 

I’ve set up two users who I’m sure, being the best of friends, will adhere to the best practices of collaborative developments. 

Tom Sawyer is first to log into the IDT in the morning.  He’s got a new measure to add to the Business Layer of our eFashion Universe so immediately also starts up a shared session which opens up the centralised Repository versions of all currently deployed models.  Now, it’s not immediately obvious how to do this and, depending on your screen size and resolution, you might have to do a little resizing of the various IDT panes. 
From the Window drop down menu, Tom selects the Project Synchronisation window which then opens up on right hand side of the workbench.
 

The session has to be opened by clicking on the session icon. If your project sync pane doesn’t have enough screen space, this icon will be hidden and you’ll have to expand the pane size. 

Opening the session connects the user to the central repository of Shared IDT Projects.  An IDT project consists of layers that can be used to create a Universe (.unx) i.e. connection layers, data foundations and business layers.  Note that each project can consist of multiple versions of each i.e. it’s possible to build more than one Universe from a single project.  They’re probably best organised through agreement within the development team.  A prime place for the collaborative working practices and standards to start being developed.

Opening up a shared project then populates both sections of the Project Synchronisation pane – the Sync Status and Shared Project lists.  From the Sync Status window we can see that the iteration of the Project uploaded on 28th February.  This reflects the iterations developed locally from converting a traditional .unv Universe into the three layered .unx Project, and then shared with the central repository. We can see that, at the moment, all are fully synchronised with the local version.



In the Shared Project section of the Project Sync pane, Tom can then ‘lock’ individual layers of the project to prevent other developers changing them whilst he works – fearlessly – on them.  An example of this is Tom wanting to lock the eFashion Business Layer so that he could add a new Measure object to it.



This will communicate to other users that this layer is currently being worked on.  When Tom’s colleague, Huck Finn, logs in that morning and opens up the Project Sync pane in his IDT workspace he sees that the Business Layer is locked.  By scrolling along the Shared Project section, Huck can see details as to which user has locked it, version number, last update date, locked date, etc…  He can also simply unlock the layer if he wants to which does rather undermine the process.  A charitable interpretation would suggest SAP are merely trying to encourage effective collaborative working through increased developer communication but, I’d suggest, this a clear candidate for project enhancement.

Anyway, Huck goes over to Tom’s desk and agrees that whilst Tom is creating this new measure, Huck will work instead on adding that new data source to the existing Data Foundation layer.  Huck checks out the Data Foundation layer and they both have a productive morning working on the same project.
After lunch, Tom is happy that his changes have been tested successfully in his local sandpit environment. He can now see, as expected, from the Project Sync pane that the version of the Business Layer on the central repository is out of sync with the version in his local sandpit environment.



By clicking on the “Save Changes on Server” icon on the top left of the Sync Status section, Tom can now check in his changed layer to the central repository.

Tom calls over to Huck and lets him know that the new Business Layer is ready for him to work with so Huck refreshes his Sync Status section and sees that, yes indeed, there is a new version there ready for him to sync to his local area with the status “Changed on Server".



Huck clicks on the “Get Changes from Server” icon and a copy of the most recent version of the Business Layer is downloaded to his local sandpit.  Huck can now test the changes he’s made to the Data Foundation that morning don’t impact on that Business Layer and proceed, fearlessly as ever, with his own development tasks.

You get the gist.  There’s a clear and straightforward, check in/check out environment available in the IDT that supports an iterative, incremental and evolutionary – an agile – approach to Universe development.  Similar functionality could be achieved in the pre-4.0 Universe Designer and a variety of CMS folders but always only for one developer at a time.  With the ‘exploded’ Universe approach, developer productivity gains are clear.  Releases can become more rapid.

There is, of course, room for improvement.  In future releases of the IDT I’d like to see a further ‘unpacking’ of the Universe so that developers can work on individual components of each layer in the same check in/check out style e.g. a Dimension Class in a Business Layer could be developed by one, whilst another worked on Measures.  It would also be nice to add some commentary to each iteration as, at the moment, Huck would have to manually check for the changes Tom had been made (assuming they weren’t working in a good collaborative fashion that is!).  And, I guess, ultimately I’d like to see full integration with a third party version control system so that, as per Collier’s recommendation in Agile Analytics, all the artefacts associated with the BI environment can be stored in one place to ensure the potential requirement to completely rebuild the environment.  Subversion would seem to be the obvious integration version control tool here given that SAP already ships it for the their BusinessObjects CMS Repository.

Not perfect then but a start at delivering the collaborative environment and version control required to support Agile Analytics.  Next time I’ll be looking at a specific feature of the IDT Business Layer development –Queries – and how they could be used to help deliver continuous testing.

Agile BI Delivery with the SAP Information Design Tool – Introduction

Over the past few years there hasn’t been a Business Intelligence event I’ve attended that didn’t cover the topic of Agile BI in one sense or another.  It was always a fascinating source of discussion that quite often raised considerable passion.  To me, it seemed a common sense approach and one that, to varying degrees, the BI projects that I have been delivering over the years have pretty much been in line with.

At the start of last year, I was browsing various BI discussion boards and came across a recommendation for a new book title “Agile Analytics : A Value Driven Approach to Business Intelligence and Data Warehousing” written by Ken Collier.  There are many wonderful things about 21st Century living but one is surely the fact that, minutes after reading these recommendations, I’d downloaded the book onto my Kindle and was fully engrossed in what Collier had to say.  I’m happy to report that the recommendations were well made and I’ve been adding my own to clients and colleagues ever since finishing it the first time round.  What I thought I’d do here is tie in some of the concepts that Collier raises in his excellent book to some of the features in the now only relatively new SAP Information Design Tool.

I’m in no way trying to tie Collier to SAP.  One of the genius points about “Agile Analytics” is that – like me - it is technology independent.  Collier makes suggestions and clearly has some tools he’s familiar with more than others but stress is made, throughout the book, that Agile BI does not rely on a single tool set.  All I’m trying to do in the series of blogs for which this is the introduction, is highlight some of the areas where SAP’s Information Design Tool (IDT) can help support Agile BI.  There is, I intend to point out, plenty of room for improvement on SAP’s part but I do believe that the IDT is a good starting point, as far as technology ever can be, for any development team considering an Agile BI approach.

Let’s start though with some words about Agile BI.  I can do no better than recommend (again) Collier’s Agile Analytics book.  The introductory chapter makes clear that Agile BI  “is not a rigid or prescriptive methodology; rather it is a style of building a data warehouse, data marts, business intelligence applications and analytics applications that focuses on the early and continuous delivery of business value throughout the development lifecycle”.  There’s a brilliant analogy in the introduction setting the scene by comparing BI to mountaineering with traditional approaches seen as Siege Style Mountaineering (large teams of climbers expending much cost, effort, sherpas, etc…) whilst Agile BI delivers a more productive Alpine Style ascents (small teams reaching the summit faster with just the bare essentials). Throughout, Collier is at pains to observe that it is “simple but not easy” and does not shy away from the challenges of Agile BI (I especially appreciated the need to be ‘fearless’ when doing the most difficult thing first to fully identify ‘project peril’. 
Another key point Collier continually makes is that delivery resource needs to perfect ‘being’ Agile rather than just ‘doing’ Agile.  It’s a state of mind rather than a strictly replicable approach. This is where his adaptation of the original Agile Manifesto comes in.  It’s worth reproducing in full:


There’s a lot more detail about this manifesto in the book.  You might well question some points of the manifesto but Collier successfully rebuts most of the obvious concerns.  It’s all good stuff but it’s not my purpose to detail it all here – go on, buy it yourself!

My focus instead is how SAP’s IDT fits into this manifesto.  Clearly it’s a ‘tool’ which is not as valued  as ‘Individuals and Interactions’ but, the key point is, it is still valued.  Collier’s book is split into two sections covering both the Management and Technical aspects of Agile BI delivery.  What I’m attempting to do in these blogs is demonstrate how the IDT can support some of those technical aspects.

Before I can do that though it might be worth reviewing exactly what the SAP IDT is.  The Information Design Tool was introduced as a development environment with SAP BusinessObjects 4.0.   In short, it’s a new tool with which developers can create the longstanding SAP BI semantic layers – the Universes.  The old Universe Designer is still there, available for developers happy with the familiar but the IDT represents a significant leap forward in functionality and, I’d argue, is much more suited to Agile BI delivery than the legacy product.  This should allow for reduced development costs and a corresponding increase in all the highlighted elements to items on the left of the Agile Analytics Manifesto e.g. where solutions are less costly to change there is more scope for responding to change in user requirements and where systems can be self documenting there is less need for comprehensive documentation.

An easy way I’ve found to understand what the IDT does is to think of it as exploding the previous development environment for SAP Universes.  Rather than working on a single entity (the .unv file), the IDT works with three distinct layers – Connection, Data Foundation and Business Layer - to produce a new Universe (a .unx file).  The diagram below illustrates this explosion using the good old e-fashion Universe imported into the new format (a straightforward enough process).


So that’s a brief introduction to both Agile BI and the SAP Information Design Tool.  Over the next three blogs, I intend to review how the latter can support the former.

i) Collaborative Development – whilst it doesn’t offer the full version controlled environment commended by Collier’s eighth chapter, the SAP IDT does provide a good advance on what’s possible for a multi-developer team on the capacity available in the traditional Universe Designer.

ii) Continuous Testing – the IDT offers an interesting, not quite complete, solution to the need to perform continuous testing on the Agile-developed Universes and I’ll also cover how using triggered SAP WebIntelligence reports can help deliver on this approach.

iii) Evolving Excellent Design – the exploded Universe layers present a model against which this can be achieved.  Correct sources can be prototyped in multiple systems, reports developed and then repointed at an optimised single source solution if required i.e. no need to redevelop reports when a new source system is delivered. 

The first SELECT FROM WHERE blog post

So, this is it, the first post on my latest foray into blogging. I've broken the habit since changing jobs over a year ago. Guess I must have missed it but what to talk about first? How to break the ice?

Let's keep it simple - why SELECT FROM WHERE?

Three reasons spring to mind:
 
1- SELECT FROM WHERE was the probably the first and most important concept I 'got' in information technology that opened up my mind to the possibility I might be able to build a career in that field and these three SQL building blocks have remained at the core of every project I've been involved with since.

2 - Perhaps the above had something to do with SFW being my initials?

3 - It was originally just going to be acronymised to SFW because it absolutely is safe for work but then I began to worry that people would start thinking there was an NSFW version somewhere and all the connotations that would bring.  Classic.  Metadata.  Issue.

OK, that seemed straightforward enough.  What about what's the aim of the blog?  In another handy three point list?

1 - SELECT interesting themes, articles and software
 
2 - FROM a variety of sources be they my personal experiences (no names here though!), other bloggers (see the list to the left), journals and books

3 - WHERE the topics are related to the management and usage of data. Initially this is going to most likely focus on issues of data quality and profiling as that's where my current career focus is but I'm not going to narrow myself to that long term.  I'm certain to go back to Business Intelligence, Reporting and Analytics at some point - that stuff is all too interesting to stay away from for long.

And that's that...the first post.  More to follow.  Thanks for reading.