One of the constant challenges for any business intelligence (BI) professional is that of balancing delivery outputs and timescales. In most BI projects the benefits analysis has identified deliverables aligned to better decision making hence the time pressures compounded by the businesses needed to improve this process. Im currently looking at ways to challenge more traditional approaches to BI develop and embrace a process of evolutionary development.
Many traditional organisations use a tried and tested method of development aligned to a sequential design and development process known as ‘waterfall’. This model has been around since at least the early 1980’s and as the name implies follows phases from one to the other like a waterfall:
The process assumes that the costs to move backward along the process are too high, hence it only flows in one direction. A typical BI project might translate the steps above into the following activities:
1 – Requirements gathering through user workshops and interviews
2 – Development of technical requirements – detail user / system analysis
3 – Develop data models and map out sources systems
4 – Develop extract, transform and load processes
5 – Constructs reporting environment including reports and dashboards
6 – Complete testing and sign off
However following this process in only one direction can often lead to batching of the activity. If we are dealing with a large project without introducing phases the batches can become large, which for anyone familiar with lean processes will know is a barrier to flow and leads to delays! The process also restricts any adjustment to approach or redevelopment of the solution as more detail is uncovered. In many cases the process steps followed lead to the development of solutions not fit for purpose and require weeks of redevelopment.
The questions to ask are ‘Why did this not get spotted earlier on in the process?’ ‘How did we miss those requirements?’ To break down the batches and improve the quality of each each phase some organisations adopt an iterative development approach. These applications of iterative development usually involve breaking each of the above steps down into a series of iterations.
– Iteration 1 : Requirements gathering through user workshops and interviews
– Iteration 2 : Development of technical requirements – detail user / system analysis
– Iteration 3 : Develop data models and map out sources systems
– Iteration 4 : Develop extract, transform and load processes
– Iteration 5 : Refine requirements gathering through user workshops and interviews
– Iteration 6 : Refine technical requirements – detail user / system analysis
– Iteration 7 : Refine data models and map out sources systems
– Iteration 8 : Refine extract, transform and load processes
– Iteration 9 : Constructs reporting environment including reports and dashboards
– Iteration 10: Complete testing and sign off
Whilst this does help to develop a better quality solution and may cut down some of the time elements it doesn’t actually deliver a useable product until all the process steps have been completed. Agile development is a methodology that BI solutions can learn from as this philosophy aims to develop solutions to users in usable chunks. For BI this is a challenge we all face as many weeks and months can go by before users see anything they can actually use day to day to make better decisions. We might spend weeks mapping out data sources and developing extraction and transformation techniques, however none of this will give the user a dashboard or report to start using now, which can lead to users losing interest…
So how can we embrace agile in BI development?
One of the key areas is to take the initial 6 steps identified earlier in a typical BI project and integrate them. In doing so individual steps can be brought closer together through use of integrated user / developer / analyst working groups to ‘story board’ solutions from ‘data to dashboard / report’. Iterations are still being used however being delivered in series of drops, where each drop represents a full cycle of development from definition to implementation i.e. At the end of drop the user will be presented with a usable dashboard or report they can start using to add value to the business.
The key elements to enabling this type of rapid BI development are:
1 – Break down the BI project into a series of drops to deliver tangible outputs to the business and the customers of the project. This may involve delivery a preliminary drop to create core foundation services needed for the rest of the project. E.g. The initial setup of an environment or data warehouse or reporting platform.
2 – Allocate defined delivery timeframes for each drop, such as 6 week outputs. This will ensure the project maintains pace and gives the business clear visibility of what will be available and when. This for me forms a key part of keeping stakeholders engaged by continually delivering something visible to the business.
3 – Streamline processes and documentation through multi disciplinary team collaboration and story boarding. Storyboards should be used to replace the traditional requirements and detailed specifications documents to capture end to end giving visibility of all activity and interactions.
4 – Keep the stakeholders involved in the process though out the development of the solution with regular review, testing and improvement.