Celona Data Migration Blog

Migrate early, migrate often

Fri, 1 Aug 2008 14:48:00 +0100

Martin Fowler's recent blog on Incremental Data Migration did a great job of highlighting the benefits of building data migration into the business-as-usual process of building or implementing new systems.

He pointed out that by starting to construct and test migration solution early and running it regularly throughout the programme you:

derisk the impact of the unknown gremlins lurking in the real data
expose the new application and users to real data rather than artifically constructed test data
raise the visibility and value of the oft-overlooked design of the migration itself

Risk is effectively reduced on all fronts because it is out in the open for all to see, and because you're using real data you greatly enhance meaningful communication between domain experts and technical staff.

I'd heartily endorse Martin's view on this from personal experience - this is definitely not an area where burying one's head in the sand is likely to be a successful strategy and getting started early is critical.

Look into the anatomy of any organisation that's been around more than a year or two and you can see its history laid down fossil-like in its layers of process and systems. And its that history (which made complete logical sense at the time it was made) around which the context has subtly shifted over time leaving packages of data that can't be made to fit the current and new functionality proposed. Good examples are old organisation structure codes, or product/tariff mappings - you can often see the outcome but the original decision logic that was applied to derive them has long-since been retired. So when you come to try to reconstruct these or move them into your new visionary architecture you don't have enough information to do it properly.

If you're fortunate these data inconsistencies will become clear early - and more likely to do so if you take an incremental approach, but will otherwise end up in the smelly pile of 'fallout' cases your migration solution can't deal with. It is crucial with this data that you can engage domain experts who remember where the metaphorical bodies are buried and can make the decision on whether to ditch or archive this data as it is no longer relevent (regulatory requirements allowing), or will be able to work out a reasonable approach to refreshing it so that it can have a meaninfgul life in the new world without breaking it.

As Martin also quite rightly points out, home-grown migration software or scripts rarely gets above the design payoff line, and is generally thrown away after it's been run once. Adding the normal enterprise software features of resilience, scalability, security, auditability and reusability are out of the question. This has to be a nonsense for any right-minded business manager - you are contemplating an event of enormous risk and impact on the precious data that drives your company and you're going to do that using throwaway, half-baked software? You certainly wouldn't entertain that approach to any other part of the business lifecycle, so why would you do that with the migration?

One of the key reasons we started Celona was because we saw the industry repeatedly taking this enormous risk, and set about the process of understanding how we could capture the many aspects - especially control and visibility, linked to great flexibility, that customers clearly needed. An industrialised approach works because it minimises risk and maximises the reuse of value added as the business evolves. That can only reside in a properly engineered built-for-purpose product - and not one that is merely adapted from another application sphere such as data warehousing, integration or virtualisation. These all have tremendous value in their own domains, but do not begin to meet the real needs of managing low-risk, orderly application data migration - and do not lend themselves easily to an incremental approach.

Levi's SAP troubles

Fri, 11 Jul 2008 12:40:22 +0100

Today's issue of The Register carries a piece on the situation Levi finds itself in taking a huge $192.5m hit on their quarterly results filing due to problems rolling out their centralized SAP ERP system.

I'm sure the teams on site trying to get this migration to fly are having a hard enough time so I will refrain from giving them further grief. I do have some thoughts that might help - in their current environment as well as future engagements.

I wonder how much the project has focused on the technology and how much on operational processes. They have 5,000 users globally who need to move to the new application and this will support key customer interactions and financial transactions for them. In other words this is actually a migration of many business processes and the people that operate them rather than merely a data migration to support a new application.

That implies a very different mindset and should drive a variety of different responses from the team.

Firstly they need to review and make sure they are applying the Four Golden Rules of Successful Data Migration (see previous blogs and articles). These will put them firmly on the front foot with the correct business engagement model.

Secondly they need to be able to separate the movement of data from the movement of applications, people and processes - this temporal abstraction gives the team time to discover problems in data and new applications early in the cycle and work around them continuing to make progress in areas that don't have problems. It also allows them to spread the processing load between source and target environments (both technical and in business operations) so that the business can contain costs of additional hardware and software to manage the transition and can buffer the impacts on their people and customers.

Thirdly they need to be able to keep everyone in the business aware of progress - they need an excellent information distribution model, and the ability to collaborate freely with all stakeholders.

My guess is that the approach and tools they are using are firmly based in an ETL paradigm and they are continually iterating round a code-trial-review-burn cycle (probably on cycle #873) and not able to show real progress with real data supporting new processes. They need third-generation approach and tooling to move on, and although I daresay they will complete the process, it will have been far more painful, expensive and damaging than was necessary.

Greener migrations

Mon, 7 Jul 2008 16:07:52 +0100

Philip Howard's latest posting at www.it-analysis.com proposes the view that real-time data integration is greener than batch because you can spread processing over more time and therefore require less additional compute power to achieve the same end.

This aligns very well with Celona's approach to progressive migration, incrementally building data as required to support the migration of business processes and people from the as-is to the to-be architecture. Load balancing the impact on network and processing capacity, and removing the need for a large staging database can radically reduce cost of ownership of transitional architectures. Last year, a very large mobile telecoms provider needed a $12m platform to support the ETL-based migration of just 1.3m customer records from the billing and CRM systems when they acquired a smaller service provider. At nearly $10 per customer record just for the hardware is a pretty high price to pay. Modern migration technologies should be able to deliver with platforms costing a fraction of these numbers - made much easier if you don't have to deliver all the data on the day of cutover.

Nothing comes for free of course, and there is a debit side in the argument which comes from the cost of synchronization. However, this can again be phased since you only need to synchronise data that is actively under control of the migration - so once it is signed off you can stop.

Reducing impact on people, business processes, software and hardware resources naturally has a direct consequence those of the planet, and I for one am happy to be part of that and I know our customers feel the same. Major enterprises are huge consumers of energy and they are increasingly subject to scrutiny by regulators and legislators on that account. Many CIOs, in looking for ways to sell transformation to their boards and investors, see the reduction in overall energy consumption as a very persuasive argument - far more compelling than describing the benefits of the latest three-letter acronym. That's all well and good, but if the transformation journey can only be delivered at huge resource cost, that will put a big damper of the case. Far better then to plump for a lean, progressive approach with reasonable costs with a heavy emphasis on reuse.

This surely is what automation is all about - creating more efficient ways of solving recurring problems? The need for Application Data Migration is growing as organizations recognize that the notion of 'steady-state' operations is rapidly dying - our enterprises are continually re-making themselves to meet new challenges both because they have to and because they can.

Don't confuse ETL with Data Migration

Wed, 2 Jul 2008 15:14:32 +0100

Just caught Johny Morris's blog in which he neatly differentiates ETL (Extract Transform Load) from Data Migration. His thesis was that ETL is a regular periodic copying of data from one environment to another, and does not therefore require any process around removing the source version of the data or decommissioning. Data Migration by contrast is the permanent, one-time movement of a set of data from one set of platforms to another, and must bundle the ability to manage the integrity of transactions from one environment to another.

To extend the point, modern application data migration has to also distinguish between migrating data and the key business processes that such data supports. ETL has no in-built capacity to do this - it generally copies operational data describing business transactions to a warehouse to support business decision making. ETL does allow data transformation to be implemented but doesn't naturally embrace the notion of abstracting the timelines for people and process to move as distinct from moving the data. Achieving this requires sophisticated, model-driven interaction with scheduling and exposure of the artifacts of that model (in particular state) to external systems and processes (best achieved through a SOA).

I'd also argue that Data Migration requires a different level of transparency to ETL - if your data warehouse doesn't get correctly loaded one day you can generally recover pretty quickly by just rerunning the ETL process again. This could be damaging, but is rarely fatal. Data Migration by contrast has the ability to completely corrupt the operational data stores and halt the key revenue generating processes in their tracks for significant periods of time. I've lived through two weeks of downtime in Billing while we unraveled an upgrade disaster and I don't want to do it again - you age very quickly! So core functions of enterprise-grade migration technology in my book must include a built-in and live audit database that clearly identifies which processes acted on which objects and when, and in a form which can drive recovery when things go wrong (which they inevitably will). This is even more important today given the levels of scrutiny on data governance required by strong regulation.

And I would also venture that the degree of complexity involved in moving data from one set of environments to another to handle operational activities is significantly greater in the application migration layer than is generally required to support BI. That goes beyond the sort of point-and-click mapping and simple transform capability embedded in most ETL products, moving towards an ontological model of the real-world environment being moved.

Are your transformations meeting Governance requirements?

Fri, 20 Jun 2008 10:14:00 +0100

Sitting in a workshop with a group of senior architects (especially those responsible for delivering enterprise structures supporting 1000s of business applications) it's always interesting to observe where a conversation heads off down an unexpected track.

Recently I was part of such a workshop discussing business transformation and data migration strategies. Of course this session was carefully planned to cover particular topics, but one branch occurred which was of particular length and intensity to be slightly unusual; a discussion about how to ensure SOX compliance during an extended business transformation.

Now it's not at all unusual to get involved in a detailed discussion about providing compliance to a regulatory regime or governance guideline, but I was a little surprised at the very real concern this group of architects had for the, also very real, business problem of ensuring correct governance during a transformation programme. This shows that technical groups are beginning to think about the three phases of business process transformation: the starting process, the target process and just as importantly the intermediate process.

For business transformations and data migrations done over the weekend, this is not an issue. There is a start process and an end process only; the business stops the old process on Friday and begins the new process on the Monday morning. But for transformations that go over weeks or months the business processes executed between the start and end points can be make or break.

One particular process issue is simply counting the company assets. In a steady state it's fairly straight forward: for example, number of customers and average revenue per customer or per user (ARPU): it 'goes up' when customers are acquired and spend more money and 'goes down' on the reverse actions. But, in the middle of the transformation - when data is constantly being transformed and migrated - business assets can be consolidated and split ad nauseam and in completely unpredictable ways.

During a weekend migration, company A (for example a Telco) starts off with 10 million customers and an ARPU of $250. After system consolidation (where multiple systems and hence customer records are merged) there may be only 8 million customers with a 'new' historic ARPU of $312. This may be good or bad - but you do know where you are. If this consolidation of customer records happens over a period of months then governance becomes much harder.

This shows a real reason and benefit for maintaining detailed and historic records of each business-object mapping between source and target systems enabling Active Reconciliation, thus providing completely transparent SOX reporting.

Are your migrations able to deliver this level of governance?

Better projects through open standards

Tue, 17 Jun 2008 10:26:00 +0100

Sitting looking out over the bay in Auckland waiting for a delayed flight, I've been passing the time thinking about how the application of standards can help achieve better project outcomes.

Standards give us a common reference point for, among other things, connecting different types of systems and technologies. Without them every time we need to transact business across multiple systems we have to define a protocol to make the conversation between the two (or more) interacting components function. That protocol tells the systems what each message means and how to handle it.

If such protocols are proprietary then only those developers in on the rules governing it can participate, which is OK if your goal is to create a closed communication environment, but results in cost and delay to the customer. Open protocols take longer to gain universal agreement and acceptance – this is natural since they require the whole community to align around a single approach to communication and may need some participants to make changes to their systems. But once adopted widely they can make a huge difference to the time, cost and energy required to integrate components.

Nokia did some work a couple of years ago that put the cost of the 'integration tax' to the Telecoms industry alone at $60bn annually. From the perspective of the systems integrator this may seem like a good thing, but the problem it creates is that this is totally non-productive in terms of innovation for the industry - its money we could be spending on doing more interesting things.

So when an organization like the Telemanagement Forum (TMF) creates an environment where the development of a whole suite of standards for thinking about the way a type of company (in their case a Telecoms Operator) functions I am inclined to believe that it is worth supporting. The TMF has created an operations framework (eTOM, or Enhanced Telecom Operations Map), a common information model (the SID, or Shared Information/Data model) as well as several other models.

I'd like to focus on the SID, since this is the piece which is most helpful to assisting in application data migration. It is important not to see the SID as a replacement for the underlying schemas supporting the major applications of CRM, Fulfillment, Billing, Service and Network Management. Typically it only describes about one third of the detail required to run these. But it is sufficient to support a higher-level business object level understanding of the main components these applications address. This makes it a good fit for what we describe in Celona as the Common Model layer - that exposes to business users enough detail for them to understand and drive a business process - in our case that of migration.

As part of a well-implemented application framework, the SID also provides the key ability to abstract from proprietary to open standards for information interchange, freeing operators from vendor lock-in and encouraging innovation. eTOM helps in that it gives people a common language and context for describing the business processes which Telcos use to service customers. Again it performs at a higher level than individual operational users will need to perform specific tasks, but that can be accommodated within implementations.

The combination of such standards, together with enlightened and trained people working with the right tools will result operators being able to update and extend their technology blueprint and their business processes independently of each other.

TMW Redux

Tue, 3 Jun 2008 11:36:00 +0100

The key focus at the recent Telemanagement World (TMW) conference was Transformation, and the message given by Keith Willetts, the TMF's Chairman was "Don't do it slowly". I don't think he meant by this that business transformation is necessarily a quick process, but rather that the time to get on and engage in it has arrived - and that was reflected in all the conversations I had at the event. Sol Trujillo, CEO of Telstra, the Australian telecoms incumbent described the transformation journey he has taken Telstra on in the past 3 years, and it's an impressive achievement and not by any means over yet. This has taken Telstra from stodgy incumbent losing market share and cash in almost every area to a dynamic leader innovating in business, culture and technology. Paul Reynolds, Telecom New Zealand's CEO is at the beginning of his journey, but plans to drive TNZ through the same step change in performance.

But for many the twin challenge of keeping business processes running while transforming to more flexible and efficient underlying operations is proving very hard. BT's Phil Dance once described this as being like trying to perform a heart transplant on a patient as he walks down the street without allowing him to bleed to death. I think this is an excellent analogy, and bears further examination. In previous times doctors might have resorted to a life-support machine while swapping out vital organs - and that's in effect what modern business migration technologies are about.

Transforming the support systems for key business processes is not just about moving application functions from one system to another, nor merely about consolidating and shifting data from one storage strategy to a more efficient one. Carrying this off successfully requires a profound engagement with the people who own and use those business processes every day, and it necessarily involves a deep understanding of data semantics and transactional timing.

Case studies presented at TMW all concluded that a structured programme is essential, with clear business goals and ownership. Managing the transformation of your business is a core activity - not something you can successfully outsource. You will however require specialist skills, knowledge and tools and a clear method - these can be bought in, but again, only within the context of that structured programme that maintains an intimate relationship between business and technologists.

The key speakers at TMW all said, and every one of the 50 or so customer meetings we were involved in reinforced the same message: the industry is at a tipping point - it is not now a question of whether operators will transform but when and how they will do so.

MDM/SOA - alphabet soup or two steps to simpler operations?

Wed, 14 May 2008 15:58:00 +0100

In recent years there’s been plenty of noise in the market around two wannabe technology waves with the usual three-letter-acronyms (TLAs) in the form of MDM (Master Data Management) and SOA (Service Oriented Architectures). Both are potentially huge with the potential to revolutionize the way we operate information technology support for our businesses. MDM offers to ensure consistency in the key product and customer records normally replicated (with greater or lesser success) across all our systems with a common maintenance application and framework, thus delivering the holy grails – a single view of the customers and the products we sell them. SOA promises to make all the data and functions of the business available ubiquitously by wrapping them in web services or the Java equivalent.

SOA and MDM could both give business important tools in simplifying the nightmarish IT complexity we’ve arrived at after 40-odd years of effort. That complexity is stifling innovation, driving very high operational costs and preventing delivery of good quality customer service. But while there are great benefits to be had, there are significant dangers as well. As part of a carefully planned and executed programme of transformation, it’s a strong strategy to wrap legacy functions in SOA capabilities and progressively transfer these to equivalent modern functions. Similarly moving to an architecture where key common data is managed through a common framework and function in a programme linked to demonstrable business benefit will yield major efficiencies. Without strong planning and clear business engagement and imperative however these technologies will go the way of so many other TLAs – great slideware, but so what?

Sensitivity to the rate at which the supported business functions can absorb change is critical in such a major transformation process. Incorporating the ability to alter direction when business conditions change will make it much easier for stakeholders to feel genuinely in control of their own destiny and thereby create real pull to help overcome the status quo. These are both common themes in any successful SOA/MDM, but given the complex nature of the problems being approached here and the need to win over many different business units to collaborate and even yield control over critical data, they will be defining. Similarly, proper governance and visibility must be provided to maximize chances of success.

With a global economic downturn in the offing (that seems without doubt now) pressure to reduce overhead and to rapidly absorb acquired operations from M&A are no-brainers – those offering SOA and MDM technologies have much to gain if they play their hands skillfully, but they must make available a clearly articulated transitional process to get businesses to their promised land.

From Netops to Softops...

Sat, 3 May 2008 16:38:00 +0100

As the Softel conference opened in London two days ago Matt Bross (BT CTO) and Bhaskar Gorti (Oracle SVP and GM) both predicted the shift from network/proprietary focused telcos to customer and community experience focused operators delivering services via software. Matt talked about an ‘innovation big bang’ driven by the combination of 10x per annum velocity on all the major technology vectors (storage, networking, compute etc) delivering a huge explosion in innovation – the net effect of which is a very great increase in complexity and movement away from convergence.

That’s a pretty interesting idea for telecoms operators who have been betting on convergence as the major driving force of their industry. But for me the new complexity will be immensely challenging to operators who have to figure out how to adapt their businesses to a landscape where the relationship between the network and software has flipped. People will not need a telecoms operator to allow them to communicate at the application layer – peer-peer already destroyed that need, rapidly relegating them to bitpipe providers. People will only engage with telcos above that layer if they deliver environments to work and play in that help them build communities to communicate, innovate and do business. That doesn’t mean a walled garden, rather it means opening the network with useful data and functions exposed as services, often for third parties to create new experiences fast.

My 8 year old daughter and her friends are obsessed with a new hybrid toy called Webkinz. So you go to the toy-store and buy a fluffy toy with a label on it that has a URL and a unique password. You take the toy home and you log onto the Webkinz website in which you meet your friends and play. From the fact that every 8-year old kid we met on our recent ski trip to France had also got a Webkinz I’m guessing this is a bit more than a north London phenomenon. That’s the kind of service people are building and telcos only equipped to deliver access and minutes have a very minimal role to play in that world. But my sense is that there is a group of leaders in the industry who are getting ready to engage in the 2.0, and have a delicate balancing act to pull off as they do so. Preserving and transforming key information about customers and the new services through the migration while the business stays online will be a determining capability they must have.

Was that a ‘success’?

Thu, 1 May 2008 17:20:00 +0100

We’ve all read about Data Migration failure and success statistics. The Standish Group have long-built a reputation for looking at IT project failures and studying the reasons for them, and last year Bloor Research and Celona published surveys on Data Migration project failure rates and the industry’s perception of Data Migration.

This all moves one to wonder “What is success and what is failure”; are we studying the right metrics? Quite reasonably, we may think, most studies focus on meeting planned IT migration delivery dates and costs. For example meta-data analysis, script development and the management of the data load; they probably include data quality analysis and cleanse activities too. These clearly represent the mass of the easily measurable project dates and costs. But are they really very useful metrics? Surely Data Migration must be measured by the success of the business in transforming to new procedures, systems and the ‘new’ data representations.

IT costs don’t really tell us very much. For example, did the business ever manage to work effectively with the new systems, how long did it take to correct data transformation errors and what costs were incurred in working around data failures on the new systems? Were any major business objectives missed, and at what cost?

So when measuring up the success or failure of data migration projects I think it would be much more useful if we consider the total cost of the business transformation project; including business process transformation. This should then be compared to the real business value that has been achieved: cost reductions, newly released revenue streams and softer factors such as improved customer and staff satisfaction. That will surely tell us how much success or failure we’re delivering, rather better than whether IT loaded the data within the planned window.

And whilst I’m on the subject of measuring true costs; my favoured search engine picked up a related blog last weekend. The writer was discussing something quite different from data migration (and particularly uninteresting as often happens) but the blog started off by talking about the fact that this was the blogger’s first weekend in weeks that he’d not toiled on a data migration project at work.

What is the cost to our business staff and developers who spend night after night, weekend after weekend completing a data migration? What is the cost to the business of this call to duty, which frankly often extends far past the original plan? It’s probably impossible to answer this, but I’m sure both the staff and their highly stressed managers are well aware that the price isn’t recovered by the after-project party and even the trip to a high profile sporting event; I was chatting to a friend recently whose firm took the project team to a Chelsea football match following his London-based company’s data migration. I had to question if this was reward for perceived project success or punishment for project failure?

DQ Matters...

Thu, 27 Mar 2008 11:07:15 +0000

Data quality has long been a slightly uneasy bedfellow for data migration - many articles have been written on the subject (just try Googling the two phrases together) and it's clear from these that the industry is some way away from real clarity on how they fit together. It is certain that a good migration project can be easily wrecked by taking the wrong approach to DQ, and a company that has enjoyed great data quality on one set of systems can destroy it completely as it migrates to new ones if it doesn't understand the issues. Since good customer service and efficient operations depend to a very high degree on good quality data it seems worthwhile to pay a bit more than lip-service to the matter.

Johny Morris's seminal work (Practical Data Migration) lays it out pretty clearly in his "Golden Rule number 3 – no organization needs, wants or will pay for perfect quality data". That doesn't mean we shouldn't aspire to improvement, simply that when faced with the practicality that it comes at a cost (either in time, money or both), businesses will invariably be ready to make a commercial decision on what quality standard they need/can afford. Even in safety-critical environments where poor data might cost lives, perfection may not be achievable given available resources, and actually getting the migration done earlier might give more benefit than delaying until every single problem has been resolved.

Data inherently contains inconsistencies and gaps – but while interestingly we assume that legacy systems and data are the major culprits, in fact they generally have had sufficient time to weed out the major problems that cause them to fail (or created work-arounds to avoid them). It's often when we take that data and try to process it in a new system environment that the issues cause failures if we allow them to. But not only do we need to decide what quality standard is appropriate, we also need to match the DQ strategy to the migration strategy. Cleansing in the source environment or in-flight seems a great idea, but we may not have the right context available to fix the problem until we get to the target in some cases. We also need to consider the ongoing strategy with respect to DQ - is this a built-in feature of the target systems? If so that would suggest we might utilize this function rather than build something additional that may not end up being consistent with that.

A well-designed approach to combining DQ with data migration can result in uncovering issues very early in the programme, and this allows business users the maximum amount of time to resolve them before they are committed to using the migrated data. A great example of this came up recently at a customer site where we were migrating a large quantity of complex network data – dry runs into a 'sandpit' environment using real data and target metadata highlighted large numbers of issues in the source data itself, the target system design, and in the transformation modeling. All were analysed and understood and the business made a series of decisions which resolved most of the issues and decided to drop other items that they could live with - traditionally these would only have been discovered in acceptance test resulting in delay and disappointment. In fact our customer is really happy because they are calling the shots and getting early business value.

My motto with regards to DQ is to build it into migration thinking from the outset, get going on it as early as possible, and make sure the right people (business folks) are making the calls on strategy, tactics and fixes. Technology's role is to appropriately guide and serve them, but can't and shouldn't do it for them.

Finally – check out http://www.datamigrationpro.com/ - an excellent new resource for data migration professionals set up by Dylan Jones.

Compliance challenge

Mon, 17 Mar 2008 17:24:00 +0000

Transformation is a huge wave running through every sector I've worked with in the past few months - including financial services, utilities, telecoms and government. The same drivers seem to apply across these areas - competition, regulation, and business leadership teams ready to take on the challenges of making their enterprises leaner and more agile. CIOs the world over are taking on the challenge of cutting through the accretion of legacy/heritage systems that have built up over the past 20 or 30 years with real resolve to get their information architectures much better aligned to their businesses. One angle that's been surfacing in conversations during the last couple of months has been the question of compliance.

It's self-evident that as we overhaul the corporate systems estate we'll need to shift and transform large quantities of data about our customers and services, and hopefully improve the quality of it along the way. And we're all familiar with the tight regulations that must be met to satisfy legal, financial and industry bodies that we're looking after our customers' data properly. Just because we're transforming doesn't generally reduce our responsibilities in any way - if anything because the process is an exception rather than business as usual, we need to be able to prove compliance throughout the process to a very high degree. Having a formal and well-tested approach to transformation with a clear, unambiguous and secure audit trail for every item we move is critical to being able to do this.

I spent a couple of hours today with a very senior information architect of a large bank, who is directly accountable for delivering the design and implementation leadership within the multi-billion dollar re-shaping programme his company has committed to. He's a guy who has bought, designed and implemented a lot of software in his career and a natural sceptic (doesn't believe that what looks great in Powerpoint necessarily works until he's seen it with his own eyes!) and has also spent a lot of weekends watching big-bang ETL migrations happen - with fingers crossed tightly. For him, the idea that there could be a less stressful, more flexible alternative was certainly appealing - but I know he will only buy into the revolution if the controls come as standard.

As with every aspect of successful data migration it's clear that we need to ensure the right subject matter experts are brought to the party and a key part of their job is to inform the team of the specific regulatory needs that the current and to-be platforms and processes have to meet, and that these are obeyed and evidenced throughout the process. Any data that is discarded must be written-off formally, error correction and transformation processes must be auditable, and reconciliations must be clear. Controls over who is allowed to access the data, whether or not it can go off-shore, how long it must be kept for have to be documented and maintained.

High-priced Hurt

Mon, 25 Feb 2008 16:19:48 +0000

I've been considering how much pain people are prepared to go through before they are willing to adopt a new approach to getting data migration done. I spoke to a manager (let’s call him Harry) at a large (very wealthy) multi-national mobile operator last year who told me his company was pretty good at getting migrations done. By example, Harry told me about a project they’d just completed where they had bought another operator with 1.3 million customers. Using traditional Extract-Transform-Load technology it had taken 13 months to move them onto Harry’s systems with a team of 130 people and cost $26m – half of which was in the leased hardware needed to completely replicate the production databases and systems (the staging area).

The industry statistics tell us that only 15% of migration projects are delivered on time and budget, so I guess Harry is justified in feeling OK with his outcome. Personally I’d say $20 per customer migrated was a pretty high contribution to net acquisition costs, and occupying 130 people was not fantastic use of their time, even on a peak team size basis – and 13 months a long time to get any value out of the process. I also suspect Harry lost a fair bit of sleep on the cutover weekend. So is Harry hurting enough to change to a progressive approach to data migration? Well maybe – his company is still doing pretty well, but as margins continually get squeezed and ARPU’s decline, spending more money and time than he needs to will come under increasing scrutiny – but my guess is he’ll probably have to see an ETL fail before he’ll jump.

And even where managers do experience major failures, it’s still hard to overcome entrenched attitudes. You’ll recall Bill from my blog a couple of weeks back where his inventory migration was struggling to complete two weeks into a 48 hour cutover window – well, I’m sorry to tell you he had to roll back the whole process and continue on the old platforms. Bill’s unfortunately so busy dealing with the fallout (blame / recrimination) that he hasn’t had time to talk to me yet, but I’m confident he’ll get there – if he’s still got his job of course!

Business DNA

Sun, 17 Feb 2008 10:58:26 +0000

Transformation is everywhere – no matter which walk of life you find yourself in there is a good chance someone is trying to change the way it works. For most organizations there are strong drivers for change, whether they are to enable growth, drive efficiency, or to respond to external stimuli. I originally trained as a Biologist and spent my university years deeply immersed in cell biology understanding how the fundamental building blocks of life continuously rearrange themselves to adapt to changing environmental conditions and competition.

We have pretty much built the major business applications we need to run our businesses now, and we know how to put them together commercially and technically. But we don’t yet have a good understanding of how to transition our businesses smoothly to use them.

The volume of data we now hold continues to grow at a vast rate, and presents a couple of really challenging issues – a lot of it is replicated many, many times and often inaccurately, and we have neither the tools nor the time to work out how to get rid of duplicates nor how to fix what is wrong. During normal operations this causes pain, but during transformation can be fatal. Funnily enough there is a similar puzzle with DNA - every broad bean cell contains 11 meters of DNA – that’s a huge amount of redundant material! Life uses the mistakes in replicating DNA to create variability and build up resilience, but unfortunately the same doesn’t work with business data – it just breaks the processes that use it.

The world of business information systems mirrors living organisms in that those that can adapt to change survive, but the consequences of failure to do so are binary. So it seems to me pretty important that we get much better at understanding how to transform the data about our businesses and our customers or we won’t be able to adapt. And we’d also better find some ways of managing the interaction between the data transformation and populating new systems (and versions of systems) from transitioning the business processes that rely on them.

Improve Your Customer Experience

Tue, 12 Feb 2008 16:02:43 +0000

I’ve been mulling over the impact a data migration can have on customer experience. Every company knows the value of happy customers today, and is just as aware of how easy it can be to lose their hard-won trust. A few months ago I heard the CFO of a UK utility company facing the wrath of the doyen of political interviewers, John Humphreys on BBC Radio 4. The utility company had been voted ‘Worst Utility in Customer Service’ and their call centers were awash with calls from irate customers. The CFO handled the interview pretty well and explained that the majority of the complaints had arisen from problems in migrating to a new billing system.

Now that sounds to me like a pretty common story – and having lived through a billing migration some years ago that went badly wrong, one that resonates for me personally. In my case, it was two weeks after go-live before we got the first customer calls querying their statements, and the first doubts started to creep in. We managed to recover after two more weeks of 24 hour shifts – not an experience I’d care to repeat.

System upgrades and replacements are very regular events, and I’ll bet the utility company’s billing upgrade met all the normal challenges – not enough investment in the vital process of connecting the business stakeholders to the project, poor understanding of the data and it’s quality, and a plan to rehearse offline several times with sample data before a big-bang cutover – leaving all the issues in the data un-confronted by the business until their customers were kind enough to find them.

It really doesn’t have to be like this now – we know better, and given how much effort has to go into acquiring customers, why wouldn’t you invest a reasonable amount of effort into looking after them while you transition between systems? Business and domestic customers are much more familiar with technology today – it’s ubiquitous in everyone’s work and home life, and they know what it’s like to upgrade systems. I wonder what the impact on customer churn was following the hashed utility billing upgrade?

To look at the positive side of the coin for a moment, I’d consider data migration a real opportunity to improve customer experience. After all, how often do you get the chance to filter and improve core customer data? If the process you’re running involves platform transformation and rationalization, this is a major opportunity to re-correlate and de-duplicate data that may have been held for many years. That can result in major savings and much greater clarity on our customer interaction.

So here’s to a more enlightened approach – one where data migration is seen as a beneficial part of projects, properly owned by the business rather than IT, where a decent value-based case is made for investment in it showing real gains in customer experience and operational efficiency, and one in which risk is properly managed from the outset rather than apologized for on public radio after the event.