For most people, in most situations, being passed a legacy is a good thing. Perhaps their long-lost uncle has left them a stately home filled with antiques. Or a legacy might be a valuable art collection given to a museum or the work of a famous author. However, in Information Technology, it is a dirty word filled with innuendo and often used as an insult. This is a strange situation and we should define exactly what we mean by legacy before addressing how to deal with it – and even if we should deal with it!
When we refer to a system as being legacy what we're really saying is that the system is built in a way that differs from how we'd choose to do so today. Legacy systems may have been written well, using the best technologies and tools available at the time, but those technologies are now out-of-date. A system written in 2001 using Java 1.2 and Oracle 8i may have made perfect sense but if you wrote it now, you'd at least use the latest versions available or even something different entirely (Scala, MongoDB, and so on didn't exist then).
Like most developers, at the center of my working life is the software development process. This might initially involve a business analysis and some support afterward, but this takes up a relatively small percentage of my time.
A software-development-centric view of the world is as follows (this is the Software Development Life Cycle (SDLC) diagram and your own process may differ slightly). There are two parts that interact with the outside world – requirements and deployment (which may be lightweight and frequent) and external interactions are wrapped in those phases.
Figure 1.1: An example software development life cycle
For some businesses (for example, public-facing websites), software is the business and there is no significant period before or after software development – it occurs constantly. These businesses are very interesting to us developers for this very reason (and therefore, these projects are often discussed at conferences); these are the exception. For most organizations, their individual IT systems perform a function that constitutes only a small part of what the organization does.
Therefore, software users view the world differently. The software development phase is a very small part of their business processes, life cycle, and lifespan. They view the world a little more like this:
Figure 1.2: The life cycle in context
Most of the processes the software users execute will originally have been done manually (I include generic software tools such as spreadsheets) and may have been done this way for 6 months to 100 years. (If you think I'm exaggerating with "100 years" then you should speak to someone who's worked for an old insurance company.)
At the end it will be decided that it is cost-effective to develop bespoke software (or spend a lot of effort configuring BPM/CRM software, and so on). This process may be iterative and deliver value quickly but will be mostly complete after a relatively short period of time (compared to the organization's lifespan).
The software is now fully live, and a software team will consider it to now be in its maintenance phase. From the organization's point of view, this is where the real value is extracted from the system. Very little money is being spent on it, but it is being used and is either helping to generate revenue or increasing the efficiency of a process.
As time goes on, there will be a decreasing number of changes made to it. Eventually, it will be considered legacy. Ultimately, the system will reach end of life and be replaced when it is no longer fit for its purpose.
If we were going to scale the diagram to indicate time, then it might look like this:
Figure 1.3: The life cycle in context and scaled with time
These are typical life cycle times that I've experienced and yours may differ – this will depend largely on the industry you work within.
Therefore, legacy should be viewed as a phase in the life cycle of the process. Note that I have a small arrow going back to Software Development from the maintenance/legacy phase, as it may have upgrades or additions many years after being deployed.
Note
Line-of-Business Systems versus Others
It was pointed out to me (thanks to Robert Smallshire) that the life cycle I've described best fits line-of-business software development and other types of systems that may vary greatly (for example, the control systems for embedded devices). This is true and reflects my personal experience but the statements on differing perspectives still hold.
I wrote a blog post covering much of this and someone commented that they refer to legacy systems as mature, as this has a more positive connotation. I think it also accurately reflects legacy as being a stage of a life cycle rather than any indication of quality.
A system maintained by professional programmers would be periodically upgraded and migrated to new technology when required, so it may never be mature/legacy. However, this is rarely a decision for programmers – it's a management decision. To be fair to management, does it really make sense to spend large amounts of money continually upgrading a system if it works? If the programmers and systems team did a good job, then no changes may be needed for many years. This indicates a high return on investment and a good quality system – a legacy!
Of course, when you do have to make changes or additions, you have a very different set of issues compared to modifying a system that is being continually developed.
Giving an example in a publication is dangerous as, in five years, it might be hopelessly out of date. However, considering the subject matter, I shouldn't make this argument. Look at the following:
Figure 1.4: © Dave Ross sourced from Wikimedia
Is this a legacy system? It's a museum piece. This is an IBM360 and was once the most popular computer on the planet. However, there are no commercially working examples left and they can only be found in museums.
This shouldn't surprise us, given the example timeline and arguments from earlier. Systems will be end-of-life or modernized eventually and it's very rare for a working system to be this old. So, what are the types of legacy systems you're most likely to come across? This will obviously change with time (and depends on the organizational areas you work within), but they are most likely to be:
Between five and fifteen years old
Without maintenance for two years
Using out-of-date APIs or technologies
If they don't meet these criteria, then you are unlikely to consider them to be legacy. Some examples are:
Java 1.2/1.3/1.4 running under Solaris 8 on an Ultra Sparc III server using Oracle 8i
C# 1.0, Windows server 2000 using SQL Server 2000
Objective C on NeXT server
J++/J#, VB6, FoxPro, Brokat Twister
A combination of spreadsheets, macros, and shell scripts
I'll cover common issues in the next section, but the preceding examples have different problems even though they are technologies from between 10 and 15 years ago. The first two (Java/Oracle and the .net stack) are very out of date but are all supported technologies. These should be simple to upgrade and support but are likely to be challenging to modify. The next two involve hardware and software that are discontinued. This will be much more challenging. Lastly, we have the very common situation of a system being bolted together from 'office tools.' In this book, I'll be covering these kinds of legacy systems.
In this section, I'll be identifying common issues with the types of systems discussed in the previous section. Depending on previous experience, these might appear obvious (you've seen it before) or surprising (you'll see it eventually). I would suggest that when you take on a new legacy project, you go through these and see what applies. Being aware of and expecting these issues will help you deal with them. I have included an Issues Checklist in the appendix to help with this.
When a system has been running for a long time, it is not unusual for supporting artefacts to be lost. The users of the system will notice if it stops working and will complain, but will anyone notice if items, such as documentation, are lost? This can happen for several reasons:
Support systems may be retired, and information removed. This can happen when there is no obvious link from the documentation to the system it concerns.
Documentation still exists but no one knows where it is or if it has been moved and the links/locations are not valid. This is a common problem when migrating document management systems.
Information is in an uncommon or unknown format. It's amazing how many files end in .doc.
Information was only ever stored locally (and the machines have been wiped) or not stored at all.
The agile manifesto says we should favor working software over comprehensive documentation, but it does not say we should not document anything. There are various types of metadata about our system that can make supporting it very difficult if missing. For example:
Usernames and passwords: The direct users of the system and its administrators may know their access details but what about the system's sub-components? What about administrator access for the database or directory server? If these are lost, then performing maintenance actions can be incredibly difficult.
Release instructions: Maybe you have the source code, but do you know how to build and release the software? Data is more likely to be kept than a running, unused service so your build and deploy server probably won't be present after 10 years. Was your build configuration in the source control system?
Last release branch details: Which one of the multiple code branches or labels was released?
Communication protocols: This is one of my personal bugbears. Many systems have been designed over the last 10 years to communicate with each other via XML messaging. Whereas developers used to document binary message formats, they've often not done so with XML as it's deemed to be 'human-readable' just because it's text-based. XML data blocks are rarely obvious, and they have so many optional elements that they are very difficult to reverse engineer. This is especially true if you only control one side of the communications, for example, receiving a message from an external source.
Licenses and other legal agreements: I'm going to talk a little more about licenses later, but can you track down the legal information about how you can use the third-party software and hardware elements in your system? They may be more restrictive than you think.
Users, including external systems: Do you know who your users are? What external systems rely on the one you control?
It is always worth tracking down and cataloguing all relevant documentation as a first step to maintaining a legacy system.
There is an overlap between lost knowledge and a lack of documentation. You could argue that no knowledge would be lost if everything was documented but this is unrealistic and incredibly time-consuming. It's likely the reasoning behind design decisions has been lost. You have the artefact (the system) but not the how and the why as to its creation.
Note
Many years ago, I worked on a system that, once a day, collected some customer information and emailed it to some users. As part of the process, a file was created (called client-credit.tmp in the /tmp folder on a Unix machine). The next day, this file was overwritten. When we began upgrading it, we moved the process into memory, so no temporary files were created. I received a very angry phone call a few days after go-live from someone whose system was broken. It turned out that another system (in a totally different part of the organization) was FTP'ing onto the server, copying client-credit.tmp from /tmp and using it. At some point, there must have been a discussion between the two departments about giving access to this information and how to do it. However, once this hack was implemented (maybe this was not supposed to be the permanent solution) it was forgotten and those responsible moved to different jobs. It was interesting that no one from either department knew this was happening.
These types of lost knowledge can create some nasty traps for you to fall into. Some of the techniques and processes discussed in later sections should help you identify and avoid them.
Also referred to as Special Voodoo, these are useful but obscure pieces of information that may either work around a bug or initiate a non-obvious function. Although not lost, it is also not documented and deliberately hidden by an individual or small group of operators. Why would anyone do this? The most common answer is job security. If you're the only person who knows how to bring up the system when it freezes or create a certain report, then it makes you much harder to replace.
This is very, very common in legacy systems and can mean that the very people you need to help you support and improve the system might be actively working against you. I'm going to be covering this a little more in a later section on Politics.
Note
I once worked for an ISV (Independent Software Vendor) that had written a complicated piece of analysis software. This had started as a simple tool and had grown organically over 10 years and was now a system used by large organizations. There was an operator for this software at one site who would 'work from home' 1 day a month in order to complete a monthly report – which he presented when he returned the next day. His colleagues (and boss) thought that he was working all day on the report, but he spent it playing golf. The report only took 10 minutes to generate by exporting the data from an obscure part of the system and then loading it into a spreadsheet template.
In a large system that performs complex functions, there is a good chance that there are huge chunks of functionality that are not used. This tends to be either due to an over-zealous development team that added features that have never been used (over-engineering) or features that are now unneeded. Often, parts of the system are replicated elsewhere in newer systems and we can think of our legacy system as being partly replaced.
Why do I describe this as a problem? Can't we just ignore these? If you don't know whether part of a system needs to be supported or maintained, you might waste a long time trying to migrate something that is not used. However, if you decide that part of the system isn't required, and then do not support or maintain it, you might later discover that it was just used very infrequently.
The owners of these rarely used features might be difficult to track down, making it hard to get a sign-off for changes.
Some unused features might still be required even if you never use them or ever intend to use them. This is actually very common in software used in a regulated or safety critical environment. For example, having the ability to delete records for security reasons (privacy regulation) or features used in disaster scenarios. You shouldn't turn off the control systems dealing with the meltdown of a nuclear power station just because it hasn't been used in 20 years. Unfortunately, there are examples of safety systems being removed in just this way – it's important you know this.
Even if your system doesn't have a potential catastrophe, you should consider whether the Business Continuity Plans and High Availability systems still work.
Figure 1.5: Old tractor image
Is this tractor used? It's old but not abandoned. It might start but is it used to do any work? Perhaps it's the only tractor in the farm that has a certain attachment to drive an essential, but rarely used, piece of equipment? Importantly, if you're the mechanic on the farm do you spend a huge amount of time keeping this working? Can you even find anyone to ask about this?
This can be due to an active decision to avoid top-down design but is more often due to a system growing in an organic way.
Many successful, large systems don't start as large systems. They start as a useful tool doing a difficult but specific task. If they do the job well, then they'll have features and options added. Over a long period of time it can morph into a large and complex system with no coherent design. Common indicators of this are:
Business logic in the incorrect place, such as in a client rather than a service. Perhaps this started as a standalone application that morphed into a multiuser application.
Multiple tools for doing the same job – for example, different XML parsers being used in different parts of the system or several ways of sending an email report. This is often due to a new developer bolting on a feature using the tools they know without looking at the rest of the system.
Inappropriate use of frameworks. For example, using threads against advice in an application server or trying to get a request-response style service to schedule tasks. This is normally due to adding a feature that was never originally envisioned and trying to do it in the current framework.
Many layers and mapping objects. New features may not fit into the current 'design' and require a lot of supporting code to work with what is there.
Sudden changes in code style – from pattern use to formatting. Where different developers add features at different points.
Some of these may be bad (time for a refactor) but for others, it may just be inconsistent. Why does this matter?
Lack of coherence in design and consistency in implementation makes a system much harder to modify. It's similar to driving in a different country – the road systems may make perfect sense, but they take a while for a foreigner to get used to. And if you move country every week, you'll never be a good driver again. Similarly, it does take a developer a while to get used to patterns and tools and they will be much less productive in an incoherent system.
It's much harder to predict the non-functional behavior in such a system. If each part of the system works differently, then performance tuning becomes hugely complex.
The first thing to note is that I've used the term fragility rather than instability. The difference is subtle, but important. An unstable system will stop working in a frequent and unpredictable way (if it was predicable then you'd have specific bugs). Whereas a fragile system will work as required but will crash horribly when a change is made.
Figure 1.6: It’s stable but fragile. Don’t open a window
Fragility can be worse than instability from a maintainer's perspective. If the users consider a system to be fragile, it can be difficult to get permission to modify it. Sometimes this fragility may be perceived rather than actual, and this is common when the system is old and no longer understood – the fear is that making any changes will stop it from working and no one will be able to repair it.
In contrast, sometimes an unstable system can work in your favor, as you can make mistakes, and no one will notice.
Later in the book, I'll be covering some of the tools and techniques for making changes to fragile systems.
In a tightly coupled system, it can be very difficult to pull apart the system's different components. This makes it hard to upgrade in a piecemeal fashion, leaving you with a stressful 'big bang' approach to upgrading and maintenance. It's also very difficult to performance-tune problematic functions.
What? Surely loose coupling is a good thing? However, what about the anecdote I told earlier about the temporary file that was FTP'd by another service? Although this is a terrible design, isn't it an example of loose coupling?
With a tightly coupled system, it's usually obvious what the dependencies are, and nothing runs without them being in a specific state. With a loosely coupled system (particularly one where services depend on each other and communicate asynchronously via a bus) this may not be so obvious. This is particularly problematic when there is a lot of unused and rarely used functionality (see earlier issues), as you don't know how or when to trigger potentially important events.
The reason these are called Zombie rather than Dead is that when I investigated some example technologies, I found out that almost no technology dies. There is always some hobbyist somewhere insisting that it's still alive, therefore, I'd count the following as zombie (dead in reality) technologies:
Technologies where the company that created and supported them no longer exists
Technologies no longer officially supported by their creators (including open source projects that have not been touched in years)
Technologies not compatible with the latest versions
Technologies where it is not possible or practical to find engineers that have any knowledge of them
Technologies where important parts have been lost (such as the source code)
I'm sure I'll get complaints about some of the preceding definitions, but these are all situations that make a system much harder to upgrade or maintain. Trying to solve a bug or modify the deployment for these can be difficult and modifying functionality might be impossible.
We should also remember that fashions and the way technologies are used can change a lot over time. Well-written Java code that used best practices from 2001 will look completely different from code a Java developer would write today.
All the components in a system, both software and hardware, will be covered by a license. This is something very often forgotten by technologists when inheriting a legacy system. Licenses can say almost anything and might put very restrictive conditions on what you do with components. This applies to everything through the entire stack and potentially even some of the data in the system if it was supplied by a third party with restrictions on its use.
Here are some questions you should ask about a system and its licenses:
Can you make changes?
Does it stop you using virtualization?
Are you not allowed to run the software on certain hardware, or does the hardware only run certain types of software?
Is reverse engineering prohibited and will this affect your ability to instrument it?
Can you modify any source you might have?
Can you even find the licenses?
Is any of the data owned by a third party?
The cost implications of not understanding licensing can be high. In the January 2013 issue of Computing Magazine, some of the implications of licensing and virtualization were discussed and the following quote was given:
Note
Hidden Licensing Costs
"If a company has SQLServer on VMware, licensed on a per core basis that can be dynamically scheduled across hosts – all machines need licenses." – Sean Robinson, License Dashboard quoted in Computing.
As well as problems regarding restricted use, licenses are also a sunk cost. If an organization has paid a large amount of money for a product, then it can be politically very difficult to change, even if the alternative has a lower overall cost.
Note
Sunk Costs are retrospective (past) costs that have already been incurred and cannot be recovered. It is an interesting area of behavioral economic research. In theory, a sunk cost should not affect future decisions – it is already spent and can't be recovered. In practice, people find it very difficult to 'write-off' previous expenditure, even if this makes economic sense.
Regulation is very similar to licensing in that it also concerns external rules imposed on a system that are outside the normal functional and non-functional requirements. Although licenses stay the same when you want to change the system, the difference with regulation is that it tends to change and force functional changes on you.
Shortly before writing this book, there have been changes to many websites in the UK. A huge number have started informing users that they store cookies and requesting permission from users to allow it. This is due to the Privacy and Electronic Communication Regulations (PECR) Act 2011 which is the UK's implementation of the European Union's ePrivacy laws. It's often referred to as The Cookie Law.
Of course, not all websites have implemented this and the ones that have are the high-volume websites with permanent development teams. Many smaller organizations with legacy websites that aren't actively developed, have not started asking these questions and are arguably not compliant. I'm not a lawyer though, so seek legal advice if you think you have a problem!
It's interesting that so many organizations have taken this regulation so seriously and it's probably because the use of cookies is so easy for an individual outside the organization to test. We can imagine someone writing a script to find UK-based websites that don't comply and then trying to sue them for infringing their privacy rights.
There are many other regulations that affect IT systems, from industry-specific ones (finance, energy, and telecoms are heavily regulated industries), to wide-ranging, cross-industry ones, such as privacy and data protection. Regulations regarding IT systems are increasing, and systems normally must comply even if they were written a long time before the regulation came into existence.
Please remember that failure to comply with regulations may be a criminal offence. If you breach a license agreement, you might find yourself being sued for a large amount of unpaid fees, but if you fail to comply with regulations (for example, money laundering or safety reporting), the consequences could be much worse.
Technology workers are often very bad at spotting political factors that affect their projects. Here are some questions you should ask yourself about your project (whether legacy, green-field, or somewhere in-between).
Who owns the project?
Who uses the project?
Who pays for the project?
Who gets fired if it fails?
Whose job is at risk... if successful?
What we should be looking out for are conflicts of interest – so let's look at the last two I've listed.
If a project fails, then the sponsors of the project and the implementers of the project will take the blame and their jobs may be at risk. This is to be expected and is one of the reasons we're motivated to do a good job and why we get harassed so much by project sponsors.
However, we rarely consider who suffers when a project is a success. It's much easier to get funding for a migration or upgrade if there is a defined business benefit; even easier still if there are increased revenues or costs savings (profit!). When an IT system saves money, it's often because jobs can be rationalized, that is, people get fired.
Note
Once, I was taken to see a client by a salesman for the software company we worked for. We walked through a room of data entry clerks (most of whom were women) and, being a stereotypical salesman, he stopped to flirt several times (he already had a couple of divorces and was working on his third). Once we left the room, he said "It's a shame isn't it?". I was a little confused and asked what he meant. His reply was "Well, once we've installed the data-importing software you've written, they're all out of a job". I hadn't been out of university for long and was naive to the real-world effects (and potential politics) of what we were doing.
Even if jobs aren't made redundant by a system improvement, the user might suffer in other ways. If tasks become standardized/commoditized, then it's much easier to replace a specific user with someone else who is cheaper (outsource the processes) or has a better attitude. Remember the examples of hidden knowledge I gave earlier? An improved system might remove this advantage for power users.
We must remember that many people simply don't like change. Technology workers are very unusual, as we like change; it's often what drew us into the job in the first place. We relish the way we work changing and making ourselves redundant probably holds no fear, as we move jobs every 18 months anyway. The users of a legacy system may have been doing the same job in the same way for 10 years and the thought of learning something new (and possibly being bad at it) can scare them.
You should also question the costs and benefits. There is a good chance that the cost is incurred by someone different to the beneficiary of the project. This is often the case in larger organizations where the IT department is separate from the business units using the software. Different solutions to issues may incur costs to different groups. For example, a re-write of a piece of software may come out of the business unit's budget, whereas an 'upgrade' might be viewed as the IT department's cost. Virtualizing an old system will probably be a cost to the IT department but the cost increase or savings on business software might fall to the end users. This will cause conflicts of interest and the decisions taken may be based on the political power of the department leaders rather than being the best decision for the organization.
Politics can be very destructive, and I've seen this as the cause of failure of many projects. From personal experience, I'd say that it's more often the cause of failure than bad technical decisions or poor implementation.
Beyond the deliberate effects of politics, there are more subtle marks that an organization can leave on the systems that are developed for it. This is described as Conway's Law (http://en.wikipedia.org/wiki/Conway's_law).
Not only are these organizational constraints reflected in the system when developed but these constraints will change over time. As people leave, teams merge or split and the barriers that existed may disappear or new ones may be created. However, the artefacts in the system will remain and build up over time. This can become obvious in long-lived legacy systems.
This affects software and hardware. Consider a system that two different departments in an organization are using. They may want separate databases (that only they have access to) and may want their processes/reports to run on servers they own (perhaps they had to provide some budget for the system's creation, and they want exclusive use of 'their' server). These departments may change dramatically over time, but you'll still have two databases and two servers.
Understanding the organizational structures of software and business teams can help to explain which quirks of the system are due to this and which are due to some large, complex subtlety that you're yet to understand (hat tip to Richard Jaques, whom I stole this sentence from).
An IT system is part of a larger process and workflow. It will have been developed to fit into the process to make it more efficient and effective. However, it doesn't just fit into the outside world; it will also affect it. The processes surrounding the system will change and evolve to work with it and its quirks. With a legacy system, these processes will have become embedded in the organization and can become very difficult to change (remember that people don't like change).
This can lead to the peculiar situation where you must get a newer, better system to work with old and inefficient processes that were originally imposed by the proceeding system. When gathering requirements for a new system, you should remember this and make sure you are finding out what is needed rather than what is currently done.
The external systems that interact with the legacy system will have also evolved to match its interfaces. This is especially true if the legacy system is quite old and the external systems were created afterwards.
This can lead to a situation where a replacement system must match old and unwanted interfaces in order to interact with dependent systems that only have those interfaces to connect to the legacy system that is now being replaced. Ancient interfaces can therefore survive long after the system they were originally designed for has been removed.
I've already described how systems can grow organically, but the data within them can also decay (The metaphor isn't perfect but I'm trying to make this interesting). This is a decrease in the overall quality of the data within the system caused by small errors gradually introduced by the users and external feeder systems. Examples include:
Data entry (typing) errors
Copy-and-paste reproduction errors such as missing the last character or extra whitespace
Old data, that is, the details are no longer true, such as an address changing
Feeder systems changing formats
Data corruption
Undeleted test data
When a system is new, there are often data quality checking tasks to keep the data accurate, but with time these tend to be dropped or forgotten (especially tasks to prune data no longer required). This is another example of something from behavioral economics, called The Tragedy of the Commons.
Note
The Tragedy of the Commons
When many people have joint use of a common resource, they all benefit from using it. The more an individual uses the resource, then the more they personally benefit. If they mistreat it or overuse it, then it may degrade but the degradation is spread amongst all the users. If an individual spends time improving the resource, then any increased benefit will also be spread amongst the whole group. From a purely logical point of view, it makes sense for any individual to use it but not spend any time improving the resource. They are acting independently and rationally (according to self-interest) but the resource will degrade over time.
These errors in data can (and sometimes without being noticed) accumulate until the entire system becomes unusable.
Did you recognize any of those issues from previous projects you've worked on? I've not listed the system being 'bad' as a problem, as it might not be perfect (all real systems have issues and bugs), but if it's a legacy system then it must have value, or it would just be turned off.
When working on a legacy system, it's worth recording issues and potential issues, such as those listed previously. Consider starting a 'Problems and Issues' document for your legacy system and recording real and potential problems. Please see Appendix 2 - Legacy Project Questions as a potential starting point for this.
I spent the last section identifying problems, but I want to be positive about a situation we often find ourselves in – and I think that some of the issues with legacy systems are due to neglect caused by negativity. So, here are a few of the reasons why working on a legacy system should be viewed as a positive experience.
I've already mentioned this but it's worth repeating. If the system had no value, then it would be turned off. It is worth finding out what the value is, even if the only reason you do so is so that you can make sure a replacement system is at least as good. Many of the positive points might only be obvious after a good investigation. Users will be very forthcoming with complaints but the parts they don't talk about can be excellent.
One of the greatest obstacles to developing a Greenfield project is knowing who the end users will be and what they require. With a legacy system, you have end users you can speak to. They can tell you what they like and dislike about the current system and tell you what they would like to be included in any upgrade. This is a massive advantage but is also one that many developers don't take advantage of.
The system might define a lot of the business and organizational processes. Many of the business decisions will be automated in the system and much of the external interaction. Some of the more complex decision processes might have been carefully defined several years before and now no one is left who knows them. The only place you can learn is by examining the IT system! I have found this to be particularly true in the finance industry. You can learn enough from an existing system to ask intelligent, relevant, and specific questions of business users (for example, the system is doing X, should it also be doing Y) rather than vague, open questions they may find difficult to answer.
If a system hasn't been modified for a while, but the way it is being used has evolved, then small changes can be made that help the users a lot. I'll talk about this more in later sections but simple performance tweaks (such as adding an index so a report runs in good time) or automating a simple, manual process can dramatically improve the user experience. You can look like a hero for little effort!
Everyone wants to work on the cool new system with a cutting-edge graphical interface on an expensive mobile device. The law of supply and demand means that the cost (pay) goes down for this popular work. A high value system that isn't trendy means you can justify being highly paid. Maybe this is shallow, but if you have a mortgage and a couple of kids, then you'll understand.
Once you have inherited a legacy system, what do you do with it and how do you approach any issues? Most books on software and system design assume it is a Greenfield project where you can build how you please. Legacy systems have constraints on what you can do and how you can do it.
This section suggests some strategies for managing these systems.
Figure 1.7: Strategy Decision Flowchart
I'm not being facetious, but I've had to include this as a possible strategy as it's the most common! If you choose to ignore your legacy system, then you should be aware that you've still made an active decision – which is to make no changes and not improve knowledge about any issues. You're not fooling anyone if you say it's on a long-term plan either.
Ignoring a legacy system is very tempting because of the issues and problems we discussed in previous sections. A lack of documentation and lost knowledge will make any project or investigation hard to even start and the politics surrounding it might make you unpopular.
Advantages
The only advantages to ignoring a legacy system is that you'll avoid any political battles in the short term and can get on with other work.
Disadvantages
The risks to ignoring the system are huge. We've already seen some of them, such as regulatory changes and zombie technologies. We also must remember that hardware, degrades and old processes might not work as expected any longer. We could argue that the biggest risk is the lost opportunity of not improving processes that could make the organization more efficient.
What is it?
Performing an investigation of your legacy system's structural components, runtime attributes and usage – that is, what makes up the system (hardware, software and configuration); how the system runs (inputs, outputs, and so on); and the use cases it is involved with.
Although this is a strategy (as you may discover that much of your system requires no further treatment or can be decommissioned), it should also be viewed as a stage to be performed before the other strategies.
Note
I worked on a project to rationalize a system that was made up of a set of components linked together by some manual processes. One process involved a user who received a file via email, which they then uploaded into an internal web application. This application performed some actions on the file and then dumped it to a central location. The users then loaded it into another system. On investigation, it turned out that all the web application was doing was changing the line endings of the file from Unix to Windows. However, the system this was loaded into would cope with either style of line endings. We had no idea why the intermediate step was first introduced (presumably, it was required at one point and then became part of the 'ritual') but it meant we could drop that part of the process and turn a sub-system off. There were good cost savings as the department was being charged a reasonable sum by centralized IT to host the web application.
You might perform a detailed investigation before deciding upon a strategy or this may have already been decided for you (we'll discuss migrations later, which are often cost- or politically-driven). Even if your strategy is predetermined, the investigation phase is important but is often passed over by keen workers desperate to show progress – this is a mistake.
Advantages
This is a good way to start working with a legacy system and shouldn't incur much political opposition as no decisions on actions are taken yet. There is also a good chance that, after an investigation, you'll change your preferred strategy – often because you've discovered value in the system you weren't aware of.
Disadvantages
You can become stuck in 'analysis paralysis' if the system is complex and opaque, so you should time-box your initial investigation. It can also be difficult to spend the required time performing the investigation properly if other commitments with higher priority repeatedly bump the work (investigations are usually ranked as a low-priority).
Advice on Implementation
An investigation should involve all the planning steps that are detailed in Part II and you should aim to generate a set of artefacts that can be used to facilitate a discussion of the system and future maintenance and upgrades.
When performing the investigation, you should also ask the following:
What is being done?
How is it being done?
Why is it being done?
Is it necessary?
Is there an alternative?
These are leading questions (looking for possible de-commissioning) but you are trying to get a deeper understanding of the system rather than just capturing it. It will also help you understand any value that it contains.
What is it?
All IT systems need maintenance, unless you intend to decommission any unused systems immediately. By maintenance I'm referring to keeping the system in its current state and correcting problems, rather than introducing functional or non-functional changes and improvements. If you intend to introduce improvements, then this would be considered upgrading, migrating, or possibly incremental improvements.
Rather than trying to improve the system, you just want to stop the system from degrading. Real systems degrade with time so monitoring and maintenance procedures are necessary to stop this from occurring. Examples of maintenance might include:
Operating system patches
Essential third-party security patches
Replacing failing hardware such as hard disks with sector errors
Removing unused users
Rebuilding indexes to maintain performance as data is modified
Deleting temporary files to stop a file system filling
You are simply trying to 'sweat the assets' and avoid any catastrophic issues. Hopefully, when the system was built, a set of maintenance actions was documented but this may not be the case; they might have been lost or evolved over time. You may require a large investigation stage, and if maintenance has been neglected, a substantial stabilization task (covered later) before resuming with regular maintenance.
Advantages
There are many benefits to maintaining a system rather than upgrading it. For example, you may work in an environment where systems require large amounts of security investigation or sign-off for functional provability. NASA is rumored to have kept systems from the 1970s running the space shuttle program because the overhead of proving reliability was too high to ever perform major upgrades or outright replacement.
Keeping a system's functionally static may also help to avoid many of the political issues that we listed in the previous section.
Disadvantages
When a system is receiving only maintenance, all the subcomponents will gradually age, as they are not being upgraded. As hardware components approach their expected lifespan, they may fail. Eventually, these hardware components will need to be replaced but replacements can be difficult to find for old systems. Eventually, an upgrade will become unavoidable, but this is best done at leisure rather than in a panic after a failure.
Software ages in a different way and it may no longer be supported, compatible, or known by new staff members.
Many projects in the maintenance phase end up morphing into other projects. It's tempting to gradually add features into a tactical system while a strategic system is being developed, which often means the strategic system never catches up.
Advice on Implementation
Scheduling investigation, stabilization, and maintenance as individual tasks will help with planning and estimation. Telling the business owner that these are separate will help with budgets and timescales (more politics). (By business owner, I'm referring to the individual or group that is the sponsor for the project. They are usually the budget holder and therefore have a lot of influence over the project. This includes organizations in the public sector and not just commercial businesses.)
You should resist allowing a business owner to 'just add a little feature' if you've agreed to maintenance only, as any scope creep will cause issues. Adding features involves another strategy, which is approached differently. I would advise getting stakeholders to agree to this strategy up-front so that, if new features are requested, you can justify extra resources.
What is it?
This strategy involves upgrading each component of a system with the latest, supported version. This may include:
Operating system (for example, Windows 2008->20XX)
Versions of Java/CLR (for example, JDK1.3->JDK1.8)
Database (for example, Oracle X -> Oracle X+1)
Processor architecture (for example, newer/faster/more processors)
Communications infrastructure (for example, 100MB NIC -> 1GB NIC)
Messaging infrastructure
Third-party products such as CRM/BPM systems
The aim is for little functional or non-functional change (although, for some third-party business-process components, this may be unavoidable).
The hardware and software development life cycle make it difficult for vendors to support old versions. Therefore, if your organization is using very old third-party software or hardware, then the original manufacturer may no longer provide support and force you into upgrading. Even if your organization wrote the main applications you are using, there will still be frameworks and infrastructure affected by this.
A strategy that involves new functionality may involve an upgrade step first.
Advantages
Upgrading current components allows users to continue using a product they are familiar with. The product should be easier to maintain as it will be better supported. Components that are upgraded to their latest versions are more likely to comply with regulations and standards, and they are often viewed as 'best practice'.
There are important cost implications for both support (out-of-date products often incur higher support costs) and in comparison, to buying new products (upgrading software should be cheaper than entirely new software and upgrading hardware components should be cheaper than an entire new system).
Disadvantages
Upgrades are rarely as simple or straightforward as they should be – here are a few issues you might face:
Dependencies between components
Performing an upgrade on a single component is often impossible. You may want to upgrade one specific component (perhaps due to known issues, such as security) but there are often dependencies between components that force you to upgrade others. These dependencies are both vertical and horizontal.
Note
Recently, I had to upgrade a piece of third-party software due to the version entering end of life. This software came with a much later version of a database driver, which didn't work well with the old version of the database we were using, so we needed to upgrade it (horizontal dependency). The latest version of the database required a lot more memory and would only work correctly in a 64-bit configuration. This required us to upgrade the operating system on the database machine (vertical dependency) and the combination of the new database and OS meant that we needed new hardware, which we virtualized to manage better.
An upgrade to a single piece of software necessitated upgrades across multiple components right down to the hardware and introduced extra layers.
Issues with data
I mentioned decaying data in the Common Issues section, and bad data creates specific problems for upgrades.
It is common for newer versions of software to assume that the data is of higher quality than was required or guaranteed by earlier versions. The operators of a system will use it in a way not originally envisioned by its creators and will insert data that is not specifically excluded. For example, adding comments to a telephone field because the software didn't enforce numeric values only. When the software vendors tested the upgrade process, they probably used sensible and expected values rather than the values found in the real world.
This unexpected data may cause repeated upgrade failures or, even worse, might create unexpected behavior in the system.
Several versions at once
If the legacy system has been in maintenance only (or ignore) mode for a while, then components of the system may be several versions behind in the upgrade cycle. It may not be possible to directly upgrade software components from your production version to the latest and you may have to move to an intermediate version (assuming you can even get hold of one). This can introduce a large amount of testing. See later sections about using virtualization to help with upgrading, as you'll want to take snapshots at these intermediate points so you can roll back to positions other than your starting point.
Sometimes, if the version you are upgrading is old enough, then you are really performing a migration and you might want to skip the intermediate versions. In this case, you would perform a fresh install and then migrate and import the data.
When you're asked to upgrade a component, you should be very careful to consider the true time required. Upgrading an item in a legacy system may be time-consuming and involve work in unforeseen areas. It can be very different to upgrading part of a non-legacy system. Please also remember that, after an upgrade, you'll still need to maintain the system – don't upgrade and forget.
Advice on Implementation
I've already listed some specific issues with upgrading and some advice for dealing with them. More generally, I would always suggest following the advice given in later sections of Chapter 3, Safely Making Changes. Just to reiterate, upgrades are always more complex and difficult than they first appear, and they need to be approached in a structured way.
What is it?
This is like upgrading, except the components are moved to a new technology or provider rather than later versions of the current technology. This can be applied to any part of the system across both the hardware and software stack. This may include:
Operating systems (for example, Solaris -> Linux)
Vendors of Java/CLR (for example, Sun JDK -> JRocket JDK)
Database vendors (for example, Oracle -> Sybase)
Processor architecture (for example, Intel -> Sparc)
Communications infrastructure (for example, Cisco -> Juniper)
Third-party products such as CRM/BPM systems (for example, Microsoft CRM -> Salesforce) and so forth.
Advantages
There are many reasons why an organization might choose to migrate rather than upgrade (or just maintain). For example, another vendor or product now has a better technical solution, or the original vendor may have ceased trading entirely. With legacy systems, it is very common for better technologies to have emerged and technology teams will want to take advantage of this.
There could be significant cost savings to be found by moving to a different product or consolidating to a single product across an organization. A good example can be found in databases where a site-wide license is purchased. There will be huge pressure to use this single product across the whole organization and stop paying maintenance fees for others.
Disadvantages
Migrations may be driven by technical or business requirements. It can be very frustrating for technical teams if the reason for migrating is cost, and equally annoying for the business owner if migrations are driven by technical desires. These kinds of conflicts are political risks and are dangerous to a project.
Many of the same issues I mentioned with upgrades are relevant to migrations but are often more severe. Problems with data aren't just due to bad data but also incompatible formats and missing, required data. You often need to enrich the data (make stuff up) to get a migration to work.
Like an upgrade, a migration may also necessitate modifications to components around a software/hardware stack.
Both upgrades and migrations will require operational changes in processes and in actions performed by the end user. The training overhead is likely to be greater for migrations than upgrades.
Advice on Implementation
There can be other drivers for the selection of technologies or requirements for migrations, such as finding suitable skills in the job market or support contract simplification. It is important for you to find out exactly why decisions are made.
The effort involved in migrating data between systems should not be underestimated. Although modifying formats can be challenging, it is missing data that will cause the most issues. You really need to make sure that these items are identified early and that you get agreement from the other stakeholders as to the resolution. It may not be possible to fill in missing data and you should make sure that you have a record of exactly what is 'made up' in order to get the system running. For example, if a new system insists that a phone number is included (and you don't have access to these) you should make sure that any 'fake' number you insert is obvious.
Lastly, please don't forget to modify any documentation to bring it into line with your upgraded/migrated system. It's very frustrating to think you have some relevant documentation only to find out that it refers to a much older version. It makes the user unsure as to whether it can be trusted or is useful.
What is it?
In this strategy, you keep the basic infrastructure and system architecture the same (probably with some upgrades or migrations first) and then either add new components or add functionality to current components. Functional additions may be driven by internal requirements to improve or expand the product or external factors such as regulation. The changes could also be driven by non-functional requirements such as coping with an increased number of users (there is a debate about what separates a functional and non-functional requirement, which we won't go into here!).
legacy systems are frequently systems that have had several incremental improvements made over their lifetime. It can be amazing to try to track down the earliest unchanged source code file, the earliest date in a comment, or the most ancient piece of hardware in a system that's been changed beyond all recognition (I was recently told that a certain modern, 3D, popular football game still had all the 2D sprites in it from a version 10 years earlier that no one had ever gotten around to removing).
Advantages
Incremental improvement allows you to give end users specific functionality on a per-requirement basis. They can see a real, defined benefit rather than lots of background improvements that make little difference to their jobs. Hopefully, you can deliver this in regular, small deployments.
Disadvantages
There is often huge pressure from business owners to "just stick it in" and get the required functionality as soon as possible (are these deadlines for an external reason or just made up?) but this leads to the dreaded technical debt. It's important to refactor but NOT JUST THE CODE BASE. If the usage changes considerably, then you might also need to change the way that software components are hosted, run, and communicate. You need to apply refactoring techniques to the frameworks and infrastructure right down to the hardware. If you are starting with hardware changes, then this might also work up through the stack as well.
Advice on Implementation
I would strongly advise you to perform any upgrades required to bring the components to their latest versions before adding any functionality. This may be opposed by the business owner if incremental improvement is viewed as a cheap option ("just stick it in"). This may be particularly true if this is driven by external factors, such as regulation, where the organization won't see any tangible benefit.
If you are writing code to add features or fix old bugs, I suggest first creating a new baseline for the code. This involves getting all the files and simply formatting them and organizing them in your preferred project structure. The new baseline should not have any functional modifications and the files should be checked into source control and labelled.
This means that any functional modifications you make from this point will show as clean and simple diffs. Without a new/clean baseline, you will find that any diff you run will include formatting and structural changes (such as file moves). You should also perform this baseline formatting on configuration files such as XML. It is amazing how inconsistent the formatting of files can become over a period, especially with many people working on them, but also how formatting fashions change.
What is it?
This is a complete re-write with no reuse beyond business knowledge. It is likely that some of the original system will be reused and, certainly, the data will be migrated. However, the intention is to replace as much as possible with a top-down approach that is, not refactoring and rewriting the system from the code upward but re-implementing from requirements down.
Advantages
This is often the preferred option for the technology team, as this gives maximum scope to use new technologies and techniques. It also avoids having to learn the idiosyncrasies of the legacy system and understand supplanted technologies. It also allows them to use familiar languages, tools, and equipment.
Disadvantages
However, you must ask yourself if you really understand everything the legacy system does or can do. We should remember the problems of lost, hidden, and implicit knowledge and understand that a replacement is very tempting but often incredibly hard to achieve. Do you really understand the business requirements or is this driven by the technology team's desires? All the issues listed for data in the Migration section are also true for a replacement.
The legacy system will have to be maintained while a replacement is being developed and new features might even have to be added due to external drivers. I have seen legacy systems that have had so much incremental improvement while a replacement was being developed, that the replacement never went live – it was constantly chasing a moving target.
Other costs to consider are those for the complete retraining of users and operations staff.
Advice on Implementation
The biggest and most common mistake people make when replacing legacy systems is to not understand all the functionality they are trying to replace. This is often because they assume the legacy system has little or no value, but this is a mistake – the system is legacy because it has value and you need to understand what this is.
You should pay attention to the data in the old system, as you will almost certainly need to import this into your new system. You don't want to get to the release stage of your replacement and realize that there is a large dataset that can't be imported or is not dealt with.
You should consider trying to run a replacement system and its legacy system in parallel configuration rather than a big-bang release. I'll go into this in more detail in the next section.
Decommissioning is the process of shutting down a legacy system. This is often combined with a 'replacement' project/strategy, as once the new system is released, then the legacy system should be turned off. This sounds simple but often isn't.
Issues
You will not want to run multiple, overlapping systems of different ages and technologies but that is a very common outcome. Often, a replacement system will not cover all the functionality of the legacy system it supposedly replaces, and the legacy system will be left running to perform a small subset of this original functionality. This means the organization has all the maintenance issues and costs of the legacy system, as well as those of the new system (a friend recently commented that he has never known a system to be fully decommissioned and his organization was filled with almost dead, zombie systems). This includes multiple teams with overlapping responsibilities, and this invariably leads to complex politics.
Advice on Implementation
You need a specific plan for decommissioning – do not just assume you can turn it off. I would suggest, at a minimum, the following steps:
Firstly, you must make sure all the stakeholders are committed to the decommissioning – there may be many hidden agendas. Please refer to the Stakeholders section.
Secondly, you need to make sure you understand all the external connections and dependencies. You are interested in external systems that are tightly coupled with your legacy system. These may need special treatment for compatibility. Please refer to the Architectural Sketches and Further Analysis sections.
Lastly, you need to decide on your actual strategy for moving from legacy to replacement systems. This may include:
The Big Bang
This involves developing and then deploying an entirely new system and turning off the old one at the same time. This has the advantage of making sure the legacy system is deactivated. However, if your analysis missed some important functionality or dependency, you might find yourself having to turn the entire system back on again. This might get repeated for multiple features until you run out of development budget – at which point, you are left with both systems forever.
You also must make sure that all users are completely trained for the new system on day one. This can be difficult and if you are forced into a rollback, there could be chaos.
The Parallel Run
This is similar to the Big Bang, in that you have developed (or possibly bought in) a complete replacement system, but you deliberately run both systems for a period of time and gradually move users and functionality from one system to the other in small increments. This has advantages in terms of user training and impact and means that a problematic feature can be individually dealt with rather than a large and embarrassing rollback.
However, it's possible (due to time or budget constraints) to not fully move the system over or miss important features. Either will force you to leave the entire legacy system running, even though only a small subsection is now being used.
Agile Component-Based Replacement
Rather than creating and deploying an entire system, the individual components within it are created, deployed and decommissioned on a one-by-one basis. This reduces any single impact and means that any unknown functionality is handled in the same way as any other known functionality. This can be unpopular with project managers, who demand a fixed timescale and cost estimate, but is more likely to deliver the functionality that users want.
It can work out much cheaper than a Big Bang release, as only the functionality that is used is replicated. You simply need to select components/functionality, replicate it in your new framework, and repeat until there is nothing left. If there is a large amount of unused functionality, you will save yourself from unnecessary re-implementation. You need to be careful of hidden or lost knowledge and track the project carefully to make sure that the legacy system is deactivated at the end. The problem here is defining what the end point is.
You might (and probably should) mix and match the suggested strategies somewhat. It is also possible to treat each component in a different way, that is, maintain some, upgrade those that need it, and replace ones that need completely new behavior. You need to understand the business' motivation and set expectations accordingly for timescales and cost. As developers, we usually favor re-writing codes –users often want little to no impact (they have jobs to get on with) and the organization's management care about cost (and, very often, the focus is on short-to-medium term costs). Choosing a strategy is difficult and involves many trade-offs.
