- Introduction
- Overall Process
- Remediation Strategies and Techniques
- Other Issues
- Technique Comparison and Concluding Remarks
Preparing for an effective transition through the year 2000
involves a great deal more than correcting problems in programs
that record dates with 2-digit years. Indeed, the process of
making the necessary code changes is often a small part of the
entire job and is eclipsed especially by the efforts necessary to
plan and manage the project and to test remediated applications
for proper operation.
Nonetheless, the technique that is chosen to remediate
existing applications often has a large impact not only on the
complexity, cost, and quality of the programming effort, but also
upon other aspects of the overall process.
A variety of remediation techniques including expansion,
windowing, compression, and encapsulation are being used to deal
with the Year 2000 problem. Further, most of these techniques can
be used on either source or object programs. Each of these
techniques brings its own benefits, challenges, and tradeoffs.
It is the purpose of this paper to review the most common and
popular of these techniques, to describe a number of ways by
which they can be applied to source or object code, to weigh
their advantages and disadvantages, and to explore their impact
on the overall Year 2000 remediation life cycle.
The opinions expressed in this paper are those of the author,
gained over a period of time participating in and observing a
number of Year 2000 remediation projects. There is no one great
solution to the Year 2000 problem. Each of the approaches
described here has been effectively used in different
circumstances. The primary objective of this paper is to provide
additional information and perspective on this subject that may
be helpful in making decisions in specific situations.
Despite the fact that techies and tool providers often begin
by arguing the religious issue of which remediation technique is
superior, the actual recoding or fixing of application
deficiencies is a small portion of the work that needs to be done
to address the looming Year 2000 problem.
In this section we will briefly discuss the overall process,
which we refer to as the "Year 2000 remediation life
cycle". A lot has been written on this subject, so our
description here will be brief and in support of the remainder of
this paper.
The Year 2000 remediation life cycle is typically segmented
into 7 phases:
- Assessment
- Planning
- Inventory
- Find/Analysis
- Remediation
- Test
- Maintenance
These phases are often not sequential. Portions of all of the
phases are normally active at any given time, and most
enterprises cycle through a number of the phases repetitively for
all or a portion of the tasks at hand.
Assessment involves investigating the overall job that needs
to be done, or the job that needs to be done in a particular
segment of the life cycle or segment of the enterprise. Many
enterprises have appointed a Year 2000 "czar" -- an
executive whose responsibility it is to guide the enterprise
through most aspects of the Year 2000 problem. This executive and
his staff assess the overall Year 2000 impact on the corporation
(not only in the IT organization), develop initial strategies to
address it, set up policies, processes, standards, and
procedures, develop budgets, and obtain funding and resources.
Assessment normally involves not only reviewing the IT assets
(programs, applications, databases) that the enterprise has
developed over the years, but also those obtained and to be
obtained from vendors and in conjunction with contractors, and
those that are outsourced. An effective assessment also includes
non-IT assets (elevators, heating systems, security, embedded
systems, etc.) and dependencies in the enterprises' supply or
production chains, including data exchanged with these sources.
Assessment is rarely a one time process. Normally, some degree
of assessment is performed, decisions are made, and operational
groups are activated. The assessment team then turns to the next
most pressing unresolved item on the list or to a newly
identified crisis. Support from the assessment team and
particularly from the assessment executive is important on an
ongoing basis. In particular, continued attention to the
strategies, policies, processes, standards, and procedures that
this group develops is essential and differentiates an effective
Year 2000 project from a bureaucratic one.
Finally, as we will see in each of the phases of the Year 2000
remediation life cycle, "triage" will become a more and
more important concept as the year 2000 approaches. Many
enterprises are not going to get the whole job done, and it is
essential that their efforts be focused on its most critical
aspects. Continued support from the assessment team is also
essential here -- what was a great strategy or remediation
technique last year, does not look so good when we realize we are
not going to get the job done in time.
The planning phase of the Year 2000 remediation life cycle
moves into the practical day to day issues that need to be
addressed to get the job done. Planning often starts at a high
level within an enterprise or one of its segments and proceeds to
the more detailed, more departmental, and more specific.
Planning and management of the plan and its identified
projects are absolutely essential to the success of the Year 2000
remediation project. Deployment of the right resources at the
right time is key. Effective and efficient use of those resources
and incorporation of concepts like "triage" are also
essential.
One of the largest and most difficult parts of the planning
process involves decomposition and segmentation of the overall
job that needs to be done into manageable units. This not only
involves resource allocation and deployment, but decomposition
and segmentation of manageable technical units of the systems to
be remediated. One of the very large problems in most enterprises
is that the IT systems are highly integrated and interdependent.
A change to one requires a corresponding change to the others.
Databases that are shared and data that flows between systems
often makes the process of segmentation very difficult to plan
and manage.
As we will see later in this paper, different technical
remediation techniques can have a substantial impact here, and
compromises often represent the best choice.
Planning must not only address how the problem is to be
segmented and fixed, it must also address how and when problem
areas are identified (impact analysis), how testing is to be
carried out, how remediated systems are to be reintegrated into
production, and how those systems are to be maintained during the
remediation time frame and afterwards. Again the technical
remediation technique chosen can have an impact on each of these
areas.
Finally, it is common in the planning stage to choose a set of
tools, techniques, and standards to be used in subsequent phases
of the Year 2000 effort.
In the inventory phase of a Year 2000 project the IT assets
are identified and grouped into manageable units. Production
systems are identified; their components are enumerated; related
source code, JCL, databases, and documentation are found and
added to the inventory. For homegrown applications, component
owners are identified. For applications obtained from vendors or
outside sources, the source is identified and if possible the
status of the asset (Year 2000 ready or not) is established.
The version number and release level of each item in the
inventory needs to be established, logged, and checked against
other components of the system (e.g., does the source code match
the load module that is used in production?). Other related
information including the source language, compiler used, compile
options, link edit options, etc. should also be obtained wherever
possible.
As the inventory of assets is established, it is often wise to
take a checkpoint and make some important decisions relating to
the remainder of the project, for example:
- Is the application necessary to the business, or can we
consider retiring it?
- Is the application critical to the business, and should
it be placed on a priority list or a "watch
list" to ensure that remediation and testing are
completed well before the deadline?
- Could we consider deferring remediation of the
application until sometime after year 2000? For example,
in some applications, Year 2000-related deficiencies are
cosmetic or their effect is only evident within the
enterprise where intelligent humans can deal with the
inconvenience or implement temporary work arounds.
- Should we remediate the application or replace it,
perhaps with a shrink wrapped product? (Note that
replacement is often a very attractive strategy, but the
cost, effort, and time required to integrate and deploy
the replacement product is often under estimated and can
ultimately create a larger problem than remediation of
the original asset).
- What is the status of vendor-supplied products -- are
they Year 2000 ready, and if not what are the
implications of moving to a Year 2000 ready version or
replacement product?
- How likely is the application to be date sensitive, and
how likely is that date sensitivity to critically impact
the application's functionality? How complex is the
application? Should it be put on a priority list, or
should fixing it be moved forward or backward in the
project schedule?
- Is the application sensitive to event horizons other than
Jan. 1, 2000? In other words, is it likely to fail before
the year 2000? When does remediation need to be complete?
- Do we have the capacity to fix the application? Do we
have the necessary domain knowledge? Do we have the
manpower? Would it be better to farm out the work? Are
outside resources available? Should we consider
establishing a retainer with an outside firm?
- Does the application involve any special problems --
source code not available, vendor gone out of business,
dependence on unique development tools or skills,
particular technical problems like dates in database
indexes, etc.?
Answers to all of these questions and more are important and
will affect subsequent phases of the project. They may also
affect the remediation strategy selected for the application and
perhaps for a whole group of applications.
In the Find/Analysis phase of the Year 2000 project,
individual applications are examined for date sensitivity, and
areas that must be fixed (remediated) are identified.
While finding dates fields in source code can be done manually
or with the use of simple tools like text editors, the process
can be greatly enhanced by using one or more automated or
semi-automated tools. There are a large number and variety of
find tools and program understanding tools on the market that can
assist in this area. A variety of techniques are used in these
tools to identify the date sensitive portions of programs that
need to be remediated:
- finding dates by field names -- often dates fields can be
identified in programs by guessing the names that
programmers have used to label those fields in source
code. Names like "current-date",
"due-date", "born",
"expiry" etc. are commonly used. Find tools are
often shipped with typical lists of names or name
fragments that can be used to search for date fields in
source code. These "seed lists" are normally
customizable and extensible by the user of the tool. Note
that many widely available find tools provide English
seed lists but fail to provide seed lists customized for
use in non-English speaking environments.
- finding dates by field format -- date data is typically
represented in a limited number of formats in programs.
Formats like YYMMDD, YYDDD are common and are often used
in situations where date computations (which must be
remediated) are performed. Other formats like MM/DD/YY
and DD-MM-YY are also commonly used but they are normally
used for display or equal comparisons and thus often
require no remediation. Some Find tools combine analysis
of field formats with seed list searching to improve
accuracy.
- literal analysis -- dates are commonly coded as literals
in programs and used in comparisons and other operations.
Most compilers provide cross reference listings that
include literals that are used in the source code. A
simple analysis of the literal cross reference list
yields not only date literals but also date variables
that are used in conjunction with them.
- using data and control flow techniques to identify dates
-- given an initial set of identified date fields,
program understanding tools can trace data and control
flow through source code to identify still other date
fields in the program and to identify statements in
source code that perform computations on date fields.
- data mining -- statistical methods can be used to blindly
analyze data files in an attempt to find dates. For
example, if we found the pattern 980325 in a file we
might suspect a date. If the next record contained
984247, however, we might suspect that the field was not
a date. While data mining using statistical methods is
not an infallible process, it can be used as a starting
point. The results it produces can be analyzed against
source code or data definitions and may provide an
initial seed list for program understanding tools. Data
dictionaries and database catalogs are other sources of
useful information.
- regression testing -- if a regression test bed exists for
the application, it may be possible to use it with the
system clock set and databases aged past the year 2000 to
look for errors or abnormal terminations resulting from
faulty date computations. Results of this kind of
testing, while not infallible, can be used before or
after remediation to uncover otherwise unidentified date
fields and program logic that requires remediation.
The techniques outlined above are best used in environments
where application source code is available and where remediation
is performed against the source code. There are cases, however,
where source code is not available and where techniques like data
mining and regression testing must be used. There are a few tools
emerging on the market that claim to be able to find dates in
object code without reference to the source. They use techniques
like looking for system calls for date data, watching control and
data flow during program execution, trapping input and output
calls and following data flow based upon known date locations or
suspected dates in data files, and trapping of program
interruptions caused by date field overflow. While these object
code analysis techniques may be the only recourse in situations
where source is not available, they are not safe or complete and
are normally used only as a last resort or in situations where
coding styles and date usage have been stylized and controlled.
In situations where source code is available, however, object
code analysis tools can be very effectively used as an additional
weapon to improve the overall analysis. Proponents of various
remediation techniques sometimes claim superiority of their
favorite methods because (among a long list of other things) it
simplifies the find/analysis phase. For example, it is said that
the expansion method is superior because it only requires you to
find and change data definition statements -- rarely do
computations or comparisons need to be changed after the data
fields have been expanded to four digit years. Unfortunately, the
devil is in the details of the Year 2000 problem. Missing one or
a few critical cases can lead to abnormal termination at random
times after year 2000, or worse to corrupted or deleted data.
Simplistic find strategies overlook the need to identify
literals, unusual coding styles, or difficult cases like using a
date or year as an index to a table. There is no substitute for
through and effective find.
A lot more will be said about remediation and about strategies
and techniques relating to remediation later in this paper in the
section entitled "Remediation Strategies and
Techniques". This section simply focuses on remediation as
one of the phases in a Year 2000 project.
Based upon decisions made earlier in a Year 2000 project,
remediation of an existing application system may take one of
three forms:
- retire the system -- it is no longer of sufficient value
to the organization to warrant fixing it. While this may
not be a typical choice for large and visible systems
within an enterprise, it is certainly a viable choice for
some smaller systems and for many ancillary programs in
larger systems.
- replace the system -- this is an alternative that often
looks very attractive at the outset. Many of the
applications currently deployed in an organization are
relatively old, and may have been scheduled for
replacement or reengineering before the Year 2000 project
began. Nonetheless, replacement can be a dangerous
strategy that normally involves a lot more work, time,
and resource than patching up the old system sufficiently
to allow it to survive a few more years. While
replacement may be a very good choice when viewed in the
long term, the immutable year 2000 deadline may make it
too high risk in a large number of situations
- fix the system -- make those changes necessary to allow
it to survive the year 2000 transition. For home grown
applications systems that support business critical and
operational processes within an enterprise, this is the
choice that is most commonly made. The focus of this
paper is on issues relating to fixing existing systems
and the remainder of this section will assume that
choice.
In the remediation phase of a Year 2000 project in which a fix
strategy has been chosen, programmers correct the Year 2000
deficiencies in individual applications using the results of the
find/analysis phase. Data definition and procedural statements in
programs are modified to correct issues relating to the use of
2-digit years in the code and in the data, and in some cases
database formats and contents are changed Remediation can be
performed manually with the normal edit/compile/debug tools used
for everyday programming and maintenance, it can be done using
automated tools, or with a combination of tools and programmer
effort. Year 2000-specific tools focused on the find/analysis
phase are common, mature, and widely used. The same cannot be
said for Year 2000-specific tools focused on remediation. While
there are a number on the market and more emerging day by day,
and while their merits are loudly proclaimed by the vendors, many
experienced users would agree that these tools are not nearly as
mature or effective as those used in the find/analysis phase.
There are a lot of program constructs like comparison and simple
arithmetic that can be handled in a straightforward and
mechanical fashion by an automated or semi-automated tool, but
there are other more difficult program constructs that crop up
regularly which require individual attention and handling by a
programmer. In most cases, a program that has been processed by
an automated or semi-automated remediation tool must be reviewed
by a human, and in many cases additional handcrafted remediation
must be performed. Most remediation tools on the market today
work with the program's source code. Recently, however, a few
tools have emerged which claim to remediate object code. These
tools use a variety of approaches some of which will be discussed
later in this paper. Remediating object code is much more
difficult in most situations than remediating source code. While
a particular object code remediation approach may appear on the
surface to provide a miraculous and low cost solution to the Year
2000 problem, experience has shown that the devil is in the
details, that the details are much more difficult to uncover and
handle in object code, and that while these tools may be able to
handle simple and straightforward cases, there are many cases
they do not handle well, at all, or safely. Safety is a very
important concept with respect to automated remediation tools for
object or source code. If a find/analysis tool does an incorrect
or incomplete job, there is good chance that the problem will be
discovered in the remediation or testing phase. If, on the other
hand, an automated remediation tool makes a bad choice, it may be
very difficult to find the problem later, and the results can be
catastrophic. Automated or semi-automated remediation tools
should leave an audit trail that can be reviewed by a programmer.
Choices the tool makes should be identified and reversible. For
example, the program statement: XXX = YEAR - 14 could be
interpreted as 14 subtracted from a 2-digit year giving a 2-digit
year 14 years earlier, or as 1914 subtracted from a year giving
the number of years since the end of World War I. Most automated
tools would probably assume the former and be correct in their
assumption. But choice may need to be reviewed and confirmed by a
programmer. Remediation is typically an iterative and interactive
process -- one that is combined with some portions of the
find/analysis phase and with the unit test component of the
testing phase. Semi-automated remediation tools often provide
interactive or iteration support that propose changes and let the
programmer review, confirm, customize, and extend the work that
is done by the tool. It is not uncommon to find tools that
perform multiple functions: aspects of find/analysis,
remediation, and in some cases, unit test. Similarly, it is not
uncommon to find programmers using a variety of overlapping tools
together in a process that combines finding, fixing, and unit
testing individual applications. Finally, as mentioned above,
this is commonly an iterative process in which fixing a portion
of the application leads the programmer to find something else
that needs to be done, and so on. One of the large problems in a
Year 2000 project is that programs and applications do not stand
alone. They are connected by process flow and data flow. One
program feeds another, and databases are normally shared between
more than one program and often between more than one
application. Making things even more difficult, copybooks,
include files, and subroutine libraries are often shared across
programs and applications. The implication of all this is that a
change to one component necessitates a change to another, and
then to another. The web of interdependencies is often very large
and very complex. Making the necessary changes to all of the
components affected and then getting them all back into
production is often a logistics nightmare. Unfortunately, there
is no easy solution to this. Project management is the hardest
part of the Year 2000 project. Many installations spend a
significant amount of time partitioning their applications into
manageable sub units and domains in an attempt to create isolated
units that can be remediated individually. One of the
technologies that facilitates this kind of partitioning is called
"bridging". "Bridges" are built between
domains or partitions, or between portions of the application set
that have been remediated and portions that have not. For example
if two applications share a database, and remediation of the
first application necessitates a change to the database, the
second application could be affected if it was not changed
simultaneously, or if a "bridge" was not built to
isolate the effect of the change in the first application.
Bridges can be built to isolate process incompatibilities and to
isolate data incompatibilities. Bridges can be external --
separate programs that are run before or between the affected
applications, or they can be internal changes to a remediated
program designed to preserve its original externals and data
formats so that interfacing applications are not affected.
External bridges are effective and commonly used to correct
process incompatibilities, but they are more cumbersome when it
comes to data incompatibilities since they typically imply that
the whole database is copied before and/or after a remediated
application is run. Internal bridges are often a simpler and more
effective way to fix data incompatibilities. To implement an
internal bridge, the logic of an affected application is modified
at or around the database I/O statements to resolve format, data
type, or content incompatibilities between the database and the
remediated application. Only records that are read or written
need to be processed by the internal bridge -- the whole database
need not be copied. There are varying opinions on what kinds of
skills are required in the remediation phase of a Year 2000
project. Clearly, the person performing the remediation or
running the remediation tool must be skilled in the art of
programming and, if working with source code, must be skilled in
the use of the specific programming language. Beyond that,
however, there are two schools of thought. One maintains that
domain knowledge of the application area and an understanding of
the business rules and policies that are imbedded in the
application are important. The other school believes that
remediation is a relatively straightforward and somewhat
mechanical job that can be performed by any decent programmer
with sufficient attention to detail. If you believe the latter,
you would make better use of the resource that has domain
knowledge and an understanding of the business rules and policies
by focusing it on other aspects of the Year 2000 project --
certainly the testing phase, and perhaps the find/analysis phase.
Or you might have your more broadly skilled programmer/analysts
oversee the work of less skilled programmers. Practically, given
the amount of work to be done and the time and resource available
to do it, it is likely that much of the remediation work will be
done in conversion factories or by programmers without domain or
business-specific skills. This argues for the need to focus those
more broadly skilled people on critical aspects of the project.
Testing, in general, is an art and not a science. In
comparison to the other phases of a Year 2000 project, there are
relatively few general, effective, and automated test tools.
While tools do indeed exist, they are typically only capable of
helping with aspects of the process and not the process as a
whole.
Testing of changes made to an existing system is normally
segmented into three phases:
- unit test -- this type of testing focuses individual
changes made to a program and is normally performed by
the programmer shortly after the change has been made
using a debugger and/or simple and specific test cases
designed to exercise the changed code. A code review or
"walk through" in which the programmer and a
peer review and discuss the changes made is another
effective technique that can be used in the unit test
phase.
- function test -- this type of testing focuses on testing
the (many) functions that an application was designed to
perform. In a well structured environment, function tests
are created to exercise all aspects of the application's
functionality identified in the application
specifications. In an ideal environment function test
cases are structured so that they can be run in an
automated environment with a minimum of human
intervention. They are collected into a regression test
bucket and run at regular intervals or whenever a change
is made to the application. New test cases are added to
the regression test bucket whenever functional changes
are made to the application or whenever a bug is
uncovered that was not caught by an existing test case.
The extensions and corrections made to an application to
support the year 2000 transition represent functional
change and should be reflected in new functional test
cases. Further, consideration should be given to
extending the existing automated test environment to
support testing in future time (e.g., across the year
2000 boundary and with data aged into the 21st century).
- system test -- in this phase of testing the application
and perhaps groups of related and interdependent
applications are placed in a simulated production
environment and run as they would be in normal operation.
The intent of system testing is to give the application
"one more try" before putting it into
production, to attempt to catch errors overlooked in
function testing, to test the interaction of
interdependent applications and the interaction between
an application and its prerequisite system components
(middleware, operating systems, etc.), and to test in a
realistic production environment (as opposed to the
typical artificial automated environment used for
function testing). While system testing is often
difficult to set up and administer, it is made more
complicated with Year 2000 since the tests should be run
not only in present time (to anticipate problems that
could occur when a remediated application is returned to
production), but also in future time -- across the year
2000 boundary and at times that could be significant to
the application.
While tools to facilitate Year 2000 testing are not nearly as
mature or common as tools designed to assist in the find/analysis
or fix phases, there are two Year 2000-specific test tools that
are widely used and very important. The first is a date simulator
that intercepts date requests made by individual applications and
returns a specified value which may be different than that in the
computer system clock. Date simulators facilitate the testing of
individual applications in future time without affecting other
operational systems running on the same computer. The second tool
is a data ager which is a utility that copies files or databases,
resetting date fields in them to a specified future date or by a
specified offset from the dates recorded in the original file or
database. These tools are considered by most to be essential in
support of function testing and regression testing. Other tools,
valuable for testing in general, are sometimes of considerable
value in the testing phase of a Year 2000 project. For example,
code coverage tools can be used to assess which portions of
applications have been exercised by test cases. In a Year 2000
environment such tools can be used to determine whether the
segments of code changed in the remediation phase have been
adequately tested. Another example involves distillation tools
that can reduce the testing effort, particularly in regression
testing, by distilling test data and databases to a manageable
size. Some organizations have spent a considerable amount of time
in the Year 2000 project planning and setup worrying about what
kind of tests should be run against a remediated system or
against an application acquired from a vendor. Which dates need
to be tested? Do tests need to be run across the boundary? Should
we test for non-dates (00/00/00) or date values that could be
used as flags (9/9/99)? This kind of discussion can consume a lot
of time and can distract planners and testers from what prove to
be the real issues and the issues that are of critical importance
in testing a remediated system. It is true that errors could
occur at the rollover, on suspicious dates, or when a date value
is used as a flag. But it is very likely that these kinds of
errors will be caught in unit test. And it is much more likely
that serious errors that cause the system to malfunction or to
corrupt data will be related things like unanticipated event
horizons, recovery from malfunctions not related to Year 2000,
side effects caused inadvertently by the programmer while he was
fixing the program, and failures related to procedural issues
(like compiling with the wrong options, or linking in the wrong
subroutine). The point is that Year 2000 testing should be
approached the same way as the testing of any significant
functional change to an operational system. The whole system
should be tested, and all of the normal test procedures and
processes should be exercised. Of course new Year 2000 test cases
should be introduced and processes should be extended to test in
future time as well as in present time. But the focus should not
be solely on Year 2000 unique aspects -- it should consider the
system as a whole and in all of its functionality in present and
future time. This said, we must admit that the discussion thus
far has assumed a well disciplined, well structured existing test
environment, and sufficient resources to extend the discipline
and structure to the testing of remediated applications.
Unfortunately, time is short, resources are scarce, and in many,
many situations the discipline, structure, test buckets, and
procedures do not exist or do not reflect the functionality of
the current system. As a result, an inadequate amount of testing
may be done and errors may surface after year 2000. Recognizing
this, it would be wise to approach testing with
"triage" in mind in the same sense that triage is used
to make decisions about other aspects of the project. Decide what
systems are business critical and plan to devote sufficient
resources to them. And plan for unexpected errors after year
2000. Have backup data, systems, and processes in place. Finally
there is great wisdom in investing in automated regression test
buckets that can be used now, between now and year 2000, and
afterwards.
Maintenance is an issue from two points of view:
- there will be a requirement to maintain and enhance
operational systems during the remediation process and
between completion of remediation and year 2000. Aspects
of this are a project management problem, and are made
more complex in certain situations where vendors or
factories are used for large scale remediation.
Furthermore, maintenance of any sort can regress a
system, particularly in cases where maintenance
programmers are not familiar with the techniques and
tools used in the remediation process. A Year 2000 clean
management process should be defined and deployed. As
part of this process, Year 2000 test buckets should be
rerun after maintenance is applied.
- The remediation method chosen can affect the complexity,
cost, and quality of future maintenance on the
application. Most agree that the expansion remediation
technique results in the clearest and most maintainable
code. Windowing and compression remediation techniques
make things more difficult because they introduce code
and constructs into the procedural portions of programs
and into business logic that is foreign to the task at
hand. This makes the code less clear and harder to
maintain. As will be seen below, compiler assisted
windowing does not suffer from this problem.
Three general and commonly used remediation techniques will be
discussed in this section: expansion, windowing, and compression.
One additional technique, encapsulation, which is not commonly
used will also be discussed because it is sometimes held out as a
"silver bullet" approach to solving the Year 2000
problem. The discussion in this section will focus on source code
remediation. Object code remediation will be discussed later.
Expanding 2-digit years to 4-digit years in programs and
databases is generally agreed to be the best solution to the Year
2000 problem. It would have been better if programmers had used
4-digit years when the systems were first implemented. Expansion
provides a clean and permanent fix (at least until the year
10,000).
Unfortunately, expansion is normally a very difficult and
costly solution to the problem because it involves changing all
of the databases in which date data resides. Not only does the
database content need to change, the format of the database must
also change, and that forces changes to any program that touches
the database, whether or not the program contains date-sensitive
logic. This involves more work than with other remediation
approaches, and more significantly, it leads to more dependencies
in the process: if we change program A and its database X,
program B that also accesses X must be changed before we put A
and X back into production. One can easily see how such a web of
interdependencies can grow exponentially and become unmanageable.
Temporary bridges can be built to isolate systems and contain the
interdependencies, but bridges take time to build and are
cumbersome to work with in an operational environment.
Note that when expansion is used, there may be a problem with
backup data and historical data that is maintained for audit
purposes. These files contain dates with 2-digit years and it may
not be possible (for legal reasons) to modify them, and even
where it is, it may not be practical. So, either special
(2-digit) versions of programs must be retained to process them
when necessary, or bridges must be built to allow them to be
accessed. Further, it is essential to maintain documentation on
the formats of this historical and backup data so that if it
needs to be accessed in future, the necessary records will be
available.
Expansion also typically requires that report formats and
input/output screen formats be changed as well. This is often
more difficult than it appears at the outset because reports and
screen formats are often carefully crafted by programmers to
maximize the value of available real estate. Further, it is also
an area where errors easily creep into the remediation process --
headings must be aligned with data or common headings and I/O
buffers may be used for multiple different purposes that are not
simple to resolve when they need to be remediated.
While it may be desirable to remediate the program and
databases to use 4-digit years, there may be a requirement to
maintain 2-digit year input for the convenience of users who have
no difficulty in understanding dates like 12/31/99 and 02/14/00.
Indeed, in most situations, humans are able to recognize and
correctly process dates that contain 2-digit years, as long as
the context in which the dates are presented to them is clear. So
it may be preferable to retain 2-digit years at least as an
option for date input and presentation.
In situations where an application is used by a limited number
of users within an enterprise, it may be possible and preferable
to choose a single, standardized, 4-digit year date format for
human input and presentation. If the application is used in a
global context where cultural issues must be addressed, or if the
application supports users outside the enterprise, support of a
variety of date formats for human usage may be required.
Consider, for example, a software vendor who markets a
spreadsheet program. Users will want to be able to input and
display dates with 2 and 4-digit years in a variety of formats.
Where 2-digit years are maintained as an option for human
input and presentation, a windowing scheme must be implemented to
support translation of those dates to 4-digit year format for
processing and storage. Many of the caveats discussed below in
the section that discusses the windowing remediation technique
are applicable to this simple use of windowing.
Proponents of the expansion technique claim that it requires
fewer changes to the logic of a program and for that reason, it
is less susceptible to errors than other techniques. There is
some truth in this claim. For example, by simply changing the
format of a data definition statement, viz.,
DUE-DATE Picture 9(6)
to
DUE-DATE Picture 9(8)
many statements involving date comparison or date arithmetic
in the procedural section of the program will work correctly
without any additional remediation. Further, because fewer
changes are made to the procedural section of the program, it
remains clearer (for future maintenance) and there is less chance
of introducing errors.
Unfortunately, it is not that simple. Changes do need to be
made to the procedural section of the application to expand
literals to 4-digit years, to ensure that headings and data
buffers are correctly constructed and aligned, to fix constructs
like indexing by a date or indexing by a 2-digit year, to correct
algorithms that convert from one date format to another, to
remove or correct special date processing where certain values
like 9/9/99 are used as flags rather than dates, to replace or
correct programming tricks that were used in the processing of
dates, and so on. Indeed, the simplicity of the expansion
technique on the surface may lead the programmer to overlook work
that needs to be done. Despite the fact than much of the
remediation work can be done by changing date data definitions,
it is important to trace and analyze the use of date data within
the procedural section of the program to ensure that all
necessary changes have been made.
Some proponents of the expansion remediation technique suggest
that the cost and complexity of the approach can be reduced by
implementing internal bridges in the programs to convert to and
from 2-digit years on files and databases to 4-digit years in the
application code. This has the real advantage of leaving the
databases and their formats unchanged and thereby reducing the
interdependency problem significantly.
Using this approach, logic is introduced into the program to
convert data read from 2-digit years to 4-digit years after each
input statement, and data to be written from 4 to 2-digit years
before each output statement. In some situations, depending upon
coding styles and standards, this may be a relatively simple
change to make in the program. It is not simple in all cases, but
it is worth investigating because, if nothing else, it can allow
an enterprise to defer changing some databases and database
formats until later in the remediation project or until after
year 2000.
This internal bridge approach does require that windows be
defined and implemented to support the date conversion. Many of
the caveats discussed in the windowing section below apply.
Further, while this approach may simplify aspects of the project,
it introduces its own complexities and costs, and does not avoid
the other code-related remediation issues discussed above.
A remediation technique that involves the use of a
"century indicator" is another variation of the
expansion strategy. A century indicator is a value that
identifies which century a 2-digit year date occurs in.
Typically, a century indicator value of 0 is used to identify the
20th century (1900-1999) and 1 to identify the 21st century,
etc. Century indicators are sometimes used as a remediation
technique, instead of using full 4-digit years because they may
be less disruptive to an existing system. For example, it is
often possible to find a spare byte in a database record format
that was not previously used and was conveniently initialized to
zero before remediation. By simply redefining this otherwise
unused field as a century indicator and changing the program to
include it in date processing logic, it may be possible to avoid
changing existing database formats or contents.
While century indicator can eliminate changes to the
database, its use normally makes program logic more complex.
Often the century indicator is a field separate from the dates it
applies to and this makes date comparisons and date arithmetic
more complex. Nonetheless, in situations where it is essential to
preserve a programmable interface or 2-digit years in a database
for compatibility purposes, use of a century indicator may be an
appropriate choice.
Century indicators have been effectively used to
remediate system code in a number of situations (viz., portions
of the MVS and AS/400 operating systems). Since system code often
defines many APIs and shared data areas whose format and content
must be preserved in support of a large number of dependent
programs, the century indicator technique provides an attractive
immediate solution. In the long term, however, it would be wise
to move away from the processing complexities associated with the
use of century indicators to a full 4-digit year approach. Some
systems that have used century indicators to preserve
compatibility for existing users also introduce new and
additional APIs that support 4-digit years and recommend their
use in new or updated applications.
The windowing remediation technique involves defining one
or more 100-year ranges or "windows", and interpreting
dates with 2-digit years in the context of the defined windows.
For example, if a window of 1950-2049 was defined, the years 50,
65, and 99 would be interpreted as 1950, 1965, and 1999, while
the years 00, 24, and 49 would be interpreted as 2000, 2024, and
2049.
The use of windows in remediation provides the
significant advantage that database formats and contents need not
be changed -- 2-digit year date formats continue to be used. The
processing logic in programs is modified to perform date
comparisons and computations correctly in the context of the
defined window(s). Obviously, the tradeoff involves the
complexity introduced into the logic of the remediated program.
Not only does this complexity complicate and slow the remediation
process, it also obscures the existing program logic and makes
future maintenance more difficult and error prone.
The fundamental advantage of windowing is that stored
data and stored data formats do not change. Thus, the problems
associated with interdependencies between applications and of
backup and historical data do not exist or are far easier to deal
with. And bridges, temporary or permanent, do not need to be
built. It is generally acknowledged that windowing is a less
expensive and time consuming remediation process than expansion
in most situations. Unfortunately, however, since in most cases
changes to procedural code in programs are more complex and more
extensive, the windowing technique can lead to the introduction
of more programming errors.
Often, when windowing is chosen as a remediation
technique, dates involved in human input and presentation (i.e.,
reports and screens) are left in 2-digit year format. In most
cases, because it is not hard for a human to correctly interpret
a 2-digit year in context, this does not cause a problem. Where
it does, dates on reports and screens should be expanded to
4-digit years.
One area that presents a problem when windowing is chosen
as a remediation technique involves the use of dates and 2-digit
years in database indexes. Many database systems (SQL, DL/I,
etc.) and indexed file systems support searching and query in the
database engine. Unless the database system has been extended to
support windows, it will not correctly handle 2-digit years as
they cross the century boundary. Two solutions are possible:
choose a different remediation technique for 2-digit year fields
that occur in database indexes (expansion or compression), or
modify the program logic to partition the queries in the two
different centuries. Either of these approaches is difficult.
Recently, a few tools have emerged to assist in this area. They
are discussed below in the section entitled "Dates in
Database Indexes". A similar problem occurs with sort
utilities. Luckily many of the commonly used sort utilities have
been extended to support sorting of windowed dates. Note,
however, that this support is not automatic in most situations.
Existing sort specifications must be corrected or extended to
specify these new parameters.
In many cases it is possible to define a single 100-year
window to be use to remediate a program. Where that is the case,
the logic changes are simpler and the chance of programming
errors is reduced. Where one or two date fields do not fit easily
into the application's defined window, it may be possible to
expand these fields to 4-digit years without a lot of difficulty
and leave the rest of the application processing 2-digit years
within a single window. In fact, it is not uncommon to find these
exceptional fields already in a 4-digit year format (e.g., birth
date, which can span more than 100 years in existing programs).
In cases where more than one window needs to be used in
an application, it should be done with care and forethought.
Rather than choosing windows indiscriminately, it is better to
choose a single "default" or "base" window
for an application or suite of interdependent applications and
then treat any additional windows that need to be defined as
clearly documented exceptions to the rule. Indeed, some
organizations have gone as far as defining an enterprise wide
default window and have mandated its use wherever possible.
Even in situations where a single window can be chosen
for an application, problems due to mismatched windows can occur
on the boundaries as data is exchanged with other applications or
with other enterprises. It is essential that the window(s)
defined for an application be clearly documented and
communicated. This is nothing new to programmers -- APIs have
always had to be documented and conformed to -- it is just one
more thing that can not be overlooked and needs to be tested.
Windows are typically defined in terms of a "pivot
year". A pivot year is a 2-digit year value that identifies
the first year in the 100-year window. If a 2-digit year in data
is greater than or equal to the pivot year, then the date occurs
in the earlier of the two centuries spanned by the window. If it
is less than the pivot year, it occurs in the later of the two
centuries. An example will clarify this: a pivot year of 50
defines a window of 1950-2049. If the year recorded in data is
50, 65, or 99, the century is 19 (because 50, 65, and 99 are
greater than or equal to the pivot year); if the recorded year is
00, 24, or 49, the century is 20 (because 00, 24, and 49 are less
than the pivot year). The concept of a pivot year is one that is
important to programmers because it simplifies the code. When
documenting an application, however, it may be simpler to specify
the window as a range, viz., 1950 to 2049.
When an application is remediated with one or more
defined and immutable windows it is said to use "fixed
windows". It is easy and superficially attractive to hard
code these window definitions into the application. But to do so
is sloppy and poor programming practice. At the very least, the
necessary pivot years should be defined as named constants that
can be easily found and changed if necessary in future.
A movable window is one whose pivot year has been
parameterized and can be changed by the programmer, the
installation, and/or the user when the system is installed or
even when it is invoked for execution. It is good programming
practice to parameterize pivot year definitions. But too much
flexibility can lead to problems. Unless there are specific
reasons to allow the pivot year to be dynamically changed it may
be best to implement the parameterization such that a recompile
or reinstall is necessary to change it. Uncontrolled modification
of a pivot year could lead to program errors and corrupted data.
One case where a parameterized and user-changeable pivot
year is a good idea is in shrink wrapped software that users use
for many different purposes. For example in a spreadsheet
program, one user might want to analyze data in the 1910-2009
window, while another might need a window from 1990-2089. In
these cases, however, it is probably best to record and process
data in 4-digit year format, while allowing the user to choose an
input/output format with 4-digit years or with 2-digit years and
a user specifiable window. Finally, even in these cases, it is
wise to choose a default window and allow the user to override it
only with an explicit command.
A sliding window is a parameterized window that is
specified as an offset from the current date and which moves
automatically as the date changes, year to year. For example, a
sliding window could be specified as 30 years in the past and 69
years in the future. In 1998, the window range would be
1968-2067; in 1999 it would be 1969-2068 -- in other words, the
window adjusts itself, or "slides" forward as the years
pass.
In certain situations, use of the sliding window
technique is advantageous insofar as it will avoid future
modification to the program, at least for a longer period of
time. Nonetheless, it should not be used indiscriminately. It is
technically more difficult to implement, it can further obscure
the processing logic in a program, and it can increase the
likelihood of programming errors. Further, unless specific
procedures are put in place to retire data as the window slides
beyond it, data corruption and other programming errors can
occur.
For these reasons, sliding windows are not as commonly
used as fixed or movable windows in Year 2000 remediation. Most
would agree that where the techniques is clearly needed it should
be used, and otherwise it should be avoided. In a large number of
cases a fixed or movable 100-year window provides more than
adequate scope for the application, and if future changes are
required, they should be toward expanded years and not toward
more complex windowing logic.
The compression remediation technique attempts to
maintain existing database record formats by reusing existing
date fields to store dates in a format that contains an
unambiguous indication of the century. Essentially, more bits are
stuffed into the same space. For example, if 2-digit years were
stored in character format in an existing database, the same two
bytes could be used to store numbers as large as 32,767 if the
representation was changed from character to binary. A variety of
compression schemes have been proposed, some of which will be
discussed below.
Compression has the advantages that database formats need
not be changed, and that unambiguous dates can be stored and
processed. It has the following disadvantages:
- in many cases dates in existing data must be changed
from their current representation to the compressed
representation
- some database systems do not allow the inclusion of
compressed data in fields set up for non-compressed data.
To accommodate compressed data, database catalogs and
dictionaries may need to be modified, and those changes
may force changes in applications that access the
databases.
- any program that touches the data must be aware of
the compression algorithm used and must employ it when
processing the date data. This includes not only
homegrown applications whose remediation is planned as
part of the Year 2000 project, but also vendor
applications that process the data. Consider, for
example, a file that is created and maintained by a COBOL
application on a mainframe computer. If that file was
downloaded to a PC and processed by a spreadsheet, both
the download program and the spreadsheet package might
need to support the compression algorithm.
- the implications of compression are probably long
term issues. It is unlikely that it will be simple or
inexpensive to change compressed data and the
applications that touch it to a better or more widely
supported format in future. Bridges may need to be built
and maintained to allow applications or packages that do
not support the compression algorithm to access stored
data that uses it.
- the compression remediation technique typically
requires changes to the data definition and procedural
sections of application programs. Those changes are often
not simple, and, like windowing in source code, tend to
obscure the business logic expressed in the underlying
program. Thus, it is likely that errors will occur in the
remediation process and in future maintenance.
- Despite these disadvantages, compression has been
used as a remediation technique in quite a number of
situations. In some cases, compression has been used on
dates in a program or file in order to deal with problems
not easily handled by other remediation techniques. For
example, in a situation where time stamps are placed in a
file and only used for sort or query purposes, it may be
smarter to use compression and retain the current file
format than to use a remediation technique that forces
changes on a lot of other applications that would not
otherwise need to be remediated. There is no
standardization in the industry on one or a small set of
compression algorithms to be used. The algorithm is
typically chosen based upon the characteristics of the
application, the platforms, the database, and the
programming language. A lot of programming tricks are
used to facilitate the process. Two broad categories of
algorithm have emerged:
- change all the date data to include a
century value. Changing date data representation
from character to binary and adding century
information is a good example of this approach.
Another takes advantage of unused bit patterns in
the sign (+/-) recorded in numeric fields to
record century information.
- leave dates in the 1900s with their existing
2-digit years, but introduce a new representation
for dates beyond 1999. For example two existing
decimal digits could be used to represent years
from 00 to 99 (i.e., 1900 to 1999), but
thereafter hexadecimal values beginning with
x'A0' represent the years 2000 and beyond. This
approach has the advantage of maintaining
existing and historical data while allowing for
the recording and processing of dates after 1999.
Note that binary sequencing of date fields is
also supported with no additional effort when
this technique is used.
Because of the disadvantages listed above, and
because the cost and complexity of using compression as a
remediation technique is typically not less than other
alternatives available, compression is not commonly used
as a source code remediation technique. It is not unusual
to find it used, however, in object code remediation.
More information on this subject will be provided below
in the section entitled "Object Code
Remediation".
One thorny remediation problem has to do with
dates that form or are a part of database or indexed file
indexes. Typically database engines and indexed file
systems provide functionality in the engine or access
method to support searching, selecting, or positioning
based upon an index value. If dates with two digit years
are made a part of these indexes, the database system or
file access method, which has no support for windowing,
incorrectly positions the years 00, 01, etc. at the
beginning of the index set and sequencing operation. This
problem has been overcome in sort utility programs that
have introduced support for a windowed year data type.
Few database or indexed file systems provide such
support, and the introduction of this kind of support
into those systems may be impractical or very difficult.
Well designed database systems like SQL support a
date data type, and store data in that format in an
expanded form that includes a 4-digit year or its
equivalent. Where this support exists and is used, this
problem does not occur. Unfortunately, it is not uncommon
for users of systems like SQL to misuse their facilities
and store dates with 2-digit years as undistinguished
numeric data.
The obvious and long term solution to this
problem is to use the correct data type where it is
available and expanded 4-digit years where dates occur in
database indexes. Unfortunately, for the reasons
discussed earlier, expansion may be difficult and costly,
and for large and complex databases, it may not provide
an option that is feasible in the time remaining to
complete remediation.
Compression has been used to address this
specific problem where expansion is not an alternative.
An algorithm that maintains the 2-digit numeric years in
existing data and a technique like hexadecimal for years
beyond 1999 is typically used for dates that occur in
database indexes. Those compressed dates may be converted
to/from another representation (like windowed dates) as
they are read and written. An internal bridge technique
can be used, for example, to accomplish this conversion
on either side of the I/O statements in the program.
With compressed dates stored in the indexes of
databases and files, the database engine or indexed file
system access method is able to use its own normal binary
sequencing algorithms to correctly perform search, query,
and position functions.
The advantage of this approach is that a
straightforward and widely used technique like windowing
can be used to remediate the application, but for just
those cases where dates occur in indexes, they can be
transformed to/from compressed dates just before or after
database access occurs. One variant of this approach
which is supported by a few tools on the market performs
the transformation between compressed and windowed dates
in a routine that sits between the program and the
database engine or indexed file access method. While
tools like these introduce their own complexities and
come with their own limitations, they offer the advantage
that an additional, unfamiliar, and complex remediation
technique does not need to be introduced into the source
code.
This use of compression to deal with the problem
of dates in database indexes appears straightforward and
very attractive at the outset. Users should be warned,
however, that many database systems are very complex, and
that application of this technique may be very difficult
in certain situations. It is advisable to enlist the help
of professionals skilled specifically in this technology
to educate, guide, and adjust the introduction of this
technique into the remediation process.
Encapsulation is a remediation technique in which
the application system is isolated, encapsulated, and run
with a simulated system clock set to a date in the past.
The technique takes advantage of the fact that the
calendar repeats itself every 28 years.
Typically, an assumption is made that the application
will run correctly in the 20th century, so the clock is
set back 28 years and the application is run in that
time. As data flows in and out of the system on screens
and in reports, dates are intercepted and shifted back or
forward by 28 years.
In one variation of encapsulation, all dates
recorded on files and in databases are aged back 28 years
during the remediation process. When this is done, stored
data becomes a part of the encapsulated system.
In the other variation of encapsulation,
dates in databases and files are treated the same way as
dates on screens and reports -- they are shifted backward
or forward when they are read or written respectively. In
this case, databases and files are considered to be
outside the encapsulated envelope in which the
application runs.
The date simulation part of this technique
is not very hard in most cases. There are a variety of
date simulation tools that were designed to assist in
testing remediated applications that can be used
effectively to simulate system dates of one's choice in
the future or in the past. These tools provide this
support on an application by application basis. It has
been suggested by some that the whole system (hardware,
operating system, middleware, and applications) could be
run with the system clock set 28 years back. Proponents
claim that this approach provides a solution not only for
application remediation but to deal with situations where
hardware and system software is not Year 2000 ready.
While it is theoretically possible for this to work in
some simple situations, it definitely will not work in
most complex environments. There are too many
non-obvious, complex date dependencies in most
sophisticated systems that could lead to subtle errors
and data corruption at unexpected times in the future.
Here are some examples of known cases why setting the
system clock back 28 years does not work:
- many PCs are incapable of representing
dates before 1980. One would expect that this
would only affect PCs, but it is not uncommon for
PCs to be used as sub components (e.g., service
processor or console) in mainframe computers.
- some widely used subsystems that
support business critical applications (like
IMS), implement integrity checks which disable
the system when time stamps are found to contain
what appears to be invalid data. Time stamps
before the product's release could be considered
invalid.
- most sophisticated systems include file
management systems which backup, recover, and
delete files using automated tools. These systems
are likely to malfunction if the clock is set
back 28 years. This can result in unexpected loss
of critical data.
- most sophisticated systems are not
stand alone -- they are connected in networks and
LANs. These connections often require a
reasonable degree of date synchronization. For
example if a file is shipped from an encapsulated
system to one that was not with a file creation
date of 1972, the non-encapsulated system might
delete it or archive it because it appears to be
very old.
Assuming that dates are simulated for
individual applications as describe above, what needs to
be done to remediate an application using the
encapsulation technique?
- if files and databases are made a part
of the encapsulated envelope, dates in them need
to be set back 28 years. Data aging tools used
for Year 2000 testing can help with this task.
Note, however, that it is not as simple as it is
in the testing environment. As encapsulated
applications are put into production, all the
data associated with them must be instantly aged
-- one day it is running in present time, the
next it is running in past time. If a problem
occurs, the files must be aged forward to restore
production to the non-remediated version of the
system. Further, we have the same problem here of
the web application and database
interdependencies as was described in the
expansion remediation section above. For these
reasons it is not common to include stored data
in the encapsulated envelope when this
remediation technique is chosen.
- if files are not included in the
encapsulated envelope, data must be intercepted
as it flows between files and programs and dates
in the records must be shifted backward or
forward. This can be done using an internal
bridge -- statements in the source program on one
side or the other of I/O requests that transform
the data before it is processed or before it is
recorded on the file. An alternative supported by
at least one vendor on the market attempts to
capture records as they flow to or from the file
access method or database engine. This is a very
difficult process, not only because it requires
sophisticated system programming tricks, but
because it requires an understanding of the
record format which may be difficult to predict
in common situations where variant records are
recorded in a file or database. It is made even
more difficult when the rules that govern variant
records are imbedded in the program and not in
the data. Both approaches described here are far
from trivial in sophisticated applications that
deal with complex databases. Both can probably be
made to work in simple situations.
- dates entered from screens or reported
on screens and reports need to be adjusted. This
is typically done with an internal bridge, but in
some cases screen maps are maintained external to
the program, and the middleware that supports
them may provide an exit in which dates can be
transformed.
- literals and other tables that are used
in date comparisons or computations must be
modified.
- algorithms that are date sensitive may
need to be modified. Algorithms that deal with
leap year are automatically dealt with by the
28-year shift. But those that treat religious
holidays are not.
- depending upon program logic, dates
values used as flags may need to be modified.
Minimally, such code should be inspected.
- if the date simulation technology that
allows the system clock to be set to a simulated
value does not support all of the specific cases
of system clock access by an application system,
internal bridges must be built at system clock
access points to adjust their values.
- the remediated application must be
thoroughly tested.
Encapsulation can be an effective
remediation technique in some simple and straightforward
cases -- cases where the design of the application
facilitates the building of internal bridges to capture
and adjust dates as they flow in and out of the
encapsulation envelope. It is an intriguing idea and one
with a lot of appeal because the technique suggests that
the cost, complexity, and time required to perform
remediation can be reduced substantially with the simple
trick of setting the clock back 28 years. Unfortunately,
in many cases, it is not that simple, and the cost and
complexity of using this remediation techniques is at
least as high as other alternatives available. Given that
tools in this area are not as mature or widely used as
those that support other remediation techniques, it is
not surprising to find that this approach is not widely
used. Finally, aspects of the encapsulation techniques
described here are proprietary and covered by patent.
Before embarking on any remediation strategy, users are
advised to ensure that they have license to do so.
Automated and semi-automated tools that
assist with source code remediation are well known and
commonly used. Recently, however, a new approach has
emerged which appears to provide significant benefits: a
number of compiler products have been extended with
functionality to assist with source code remediation. In
a typical implementation, the language (e.g., COBOL)
which the compiler supports is extended with syntax that
allows date fields defined in a program to be identified
and described. The programmer working on the application
remediation uses tools and techniques of his choice to
identify the date fields and then "annotates"
those field definitions in the source code with
information that identifies them as dates and describes
the specific date format(s) used. The programmer then
specifies a remediation technique (e.g., fixed windowing)
and parameters like the pivot year. In many cases, no
additional modifications to the program are necessary. As
the program is compiled by the enhanced compiler, date
usage is recognized in procedural statements, and the
compiler generates appropriate object code to correctly
handle the dates in the context of the specified
remediation technique.
For example, in a COBOL program which
defines:
02 CURRENT-DATE PICTURE 9(6)
02 DUE-DATE PICTURE 9(6)
The programmer modifies the definitions to
include identifying and format information:
02 CURRENT-DATE PICTURE 9(6) DATE-FORMAT
YYMMDD
02 DUE-DATE PICTURE 9(6) DATE-FORMAT YYMMDD
and specifies, through the use of compiler
directives, that remediation with fixed windows and a
pivot year of 1950 are to be used. Subsequently, when the
compiler recognizes the use of these date fields in
comparisons or certain types of arithmetic operations, it
generates correct code to perform the computations in the
context of the specified window.
The obvious advantage of this approach over
other tools that assist with source code windowing is
that little if any of the procedural section of a program
needs to be modified to support windowing. The remediated
program is far clearer and easier to maintain since it is
free of the clutter of windowing logic in its procedural
section. A second obvious advantage of this approach is
that far less code needs to be changed, and where changes
are required they are simpler and more straightforward.
This leads to fewer errors introduced in the remediation
process.
This approach offers many of the benefits of
expansion (clean code, simple changes) without the
disadvantage of having to modify databases, screens, and
reports, and without introducing the interdependency
problem inherent in the expansion approach.
Some compilers that have implemented this
function support a variety of date formats that are used
in computation and comparison (e.g., Gregorian dates,
Julian dates, year fields), and they support both 2-digit
years and 4-digit years in those date formats. Since the
compiler knows that annotated fields are dates, it is
able to support the combination of 2-digit year and
4-digit year formats in same procedural statements. This
allows individual fields in an application to be
remediated using expansion or windowing, depending upon
specific needs (e.g., one field contains dates that could
span more than 100 years, but all the rest can be
windowed). It also supports incremental movement to the
"ideal" expansion solution.
Finally, other aspects of the compiler's
technology can be used advantageously with these
extensions. Since the compiler has extensive syntactic
and semantic knowledge of the source program it is able
to generate diagnostic messages to warn the programmer of
errors (e.g., incorrect or conflicting use of a date
field, suspected missing annotations), or of code that
could involve ambiguous computations. The same facilities
can be used to produce messages that provide an audit
trail of all of the choices made by the compiler and of
the effects of the remediation it has produced in
response the annotated field definitions. Similarly the
compiler can generate debug hooks on statements that
involve dates to be used to facilitate the debugging and
unit test processes.
While these compiler extensions provide
significant advantage and savings in the remediation
effort, they do not handle all cases. And in fact,
because compilers insist on "safe" and
predictable program constructs, in some cases they
require the programmer add to additional specifications
to the body of a program to resolve ambiguous situations.
A few examples illustrate these points:
- compiler assisted remediation will not
handle many cases of indexing a table with a
2-digit year, because in many situations the
table needs to be extended to handle years beyond
1999.
- the compiler may refuse to compare a
date field with a non-date field, because in most
cases this is an error (the programmer forgot to
identify the second field as a date).
- the compiler may refuse to add what
appears to be a 4-digit year to a 2-digit year
(e.g., 97 + 1900, a computation which converts a
2-digit year to a 4-digit year in limited cases)
because in many cases these computations will
yield incorrect results after 1999.
- the compiler does not support dates in
all forms of mathematical computations because in
many cases requests for such computations are
erroneous, or require algorithmic changes beyond
the scope of simple windowing.
While the compiler does not handle these and
some other unusual program constructs, it does identify
them to the programmer where they occur so that
appropriate source changes can be inserted into the
program manually. In many cases the source changes are
simple or trivial and are supported by a set of new
date-related intrinsic functions that are supplied in the
compiler library. Compiler assisted remediation appears
to offer significant advantage over manual methods and
over other tools that assist with source code
remediation. Why then is this technology not more widely
used? The answer is unfortunately simple -- it was only
recently invented and has only appeared in selected
products in late 1997 and early 1998. In situations where
it meets an organization's needs in terms of platforms,
languages, and compiler versions supported, and where the
organization has sufficient remediation work yet to be
completed, this technology should be seriously
considered. Note that because it supports both sliding
and fixed windows, compiler assisted remediation can
coexist with other remediation tools that support
windowing, and can be used in situations where portions
of the application inventory has already been remediated
using another tool.
Almost all remediation projects performed to
date have used source code remediation techniques. The
reason is obvious: object code remediation is often very
difficult technically, and the probability of finding and
fixing all Year 2000-related problems in object code is
lower than when source code remediation is used. Until
very recently there have been no viable object code
remediation tools on the market.
The appeal of object code remediation tools
is seductive: you do not have to touch your program, the
"magic" tool will do the job for you
transparently and without effort on your part. The truth
is that object code remediation is a difficult job and a
risky one even with the best tools on the market today.
And even if successful, it leaves the enterprise with
applications that are extremely difficult to maintain and
enhance in future. So why consider it at all?
- there are cases where object code
remediation is the only alternative short of
rebuilding the application from scratch. The
source code may be lost or the vendor from whom
the application was acquired may have gone out of
business.
- while difficult and risky, object code
remediation in some cases could be a lot faster
and consume a lot less resources than source code
remediation. If nothing else it could provide a
stop gap measure to defer a portion of the work
until after year 2000.
Before embarking on a project using this
approach, consider the following:
- there are very few credible object code
remediation tools on the market. Choose a vendor
with credibility and demonstrated experience in
the area.
- object code remediation is more likely
to be successful in simple situations than in
complex ones. Choose segments of the application
inventory that are technically simple, e.g.,
batch COBOL with simple file structures and
simple business logic.
- choose non-critical applications to
remediate -- you can not afford to fail when it
comes to business critical applications.
- you are more likely to be successful if
your installation has clear and established
coding standards and stylized code in the
applications. For example, if all dates use a
specific format and data type (e.g., YYDDD and
packed decimal) and if you can find an object
code remediation tool that handles those
standards and styles, there is a much higher
probability of success.
- schedule time and resources after year
2000 to replace the application or to remediate
its source code. Object code remediation is a
poor choice for the long term
Object code remediation tools use a variety
of approaches to solve the Year 2000 problem. The first
we will discuss is encapsulation. The encapsulation
technique uses a simulated system clock set back 28 years
to cause the system to appear to continue to run in the
20th century and thus not to have to deal with the date
rollover problem. An object code remediation tool that
uses this technique must capture date data as it flows
into or out of a program and adjust the dates backward or
forward by 28 years. In some simple environments this may
not be hard to do. If all I/O flows through redirectable
pipes as is common in simple UNIX and C programs and if
dates are in easily recognizable locations or formats, it
may not be hard to intercept them as they flow in and out
of the system. In other more sophisticated or complex
cases, it may be far more difficult. In particular, the
issues described earlier involving variant records and
complex file or database interfaces are hard to deal with
when this remediation technique is attempted.
In addition to catching dates in program
I/O, any dates coded as literals or constants in programs
which are used in computations or comparisons, must be
identified, adjusted, and patched back into the object
program. A tool might scan an object program or
disassemble it looking for what looked like dates or two
digit years and adjust those. But that could incorrectly
adjust dates used in report headers and the like, or
other constants that merely happened to look like a date
or 2-digit year. This whole process is made more complex
because most compilers will generate a single constant or
literal even if it is used for multiple purposes. So, the
constant 97 may be a year used in a computation, a year
used in report headings, and a numeric constant used for
unrelated purposes. In such a situation changing it would
result in incorrect behavior in the program. There is not
easy or obvious fix for this problem, but the problem may
not occur in many programs.
Programs that contain date sensitive
algorithms may pose a problem with this remediation
technique. The algorithms may have to be adjusted to
reflect the fact that the system clock has been set back,
and this is not normally possible to do in object code.
It should be obvious that object code
remediation using encapsulation is not a technique that
can be used in general or complex situations. It may work
and may provide benefit in individual and simple
situations. Where it is attempted, more effort must be
put into exhaustive program testing to ensure that things
have not been overlooked or incorrectly remediated.
Finally, before embarking on a project that uses this
technique, the risks should be evaluated. You could end
up in a much worse situation if the technique does not
work and your are left with an unremediated application
and less time left to fix it.
The compression technique stuffs more bits
into an existing space and thereby includes a positive
indication of century in recorded dates. The most
effective compression techniques are those that preserve
00 to 99 for the years 1900 to 1999 and use a non-numeric
representation (e.g., hexadecimal) for years after 1999.
The machine language instructions on certain platforms
that are designed to process numeric data may refuse to
handle non-numeric hexadecimal bit patterns in numeric
computations. When confronted with what appears to be
invalid data the instruction may cause a program
interrupt. This is the case for most instructions that
perform comparisons or computations on packed decimal
data on System/390. Packed decimal is a common numeric
format on System/390 used to process and store date data.
Using this remediation strategy a tool
intercepts dates as they flow from screens into the
program and transforms years after 99 into this
compressed pseudo-hexadecimal format. Thereafter, when
this data is processed by packed decimal instructions, a
program interrupt occurs. The remediation tool provides
an interrupt handler which is given control when the
interrupt occurs and which simulates valid operation of
the interrupted instruction (perhaps returning a result
in this compressed format) and then returns control to
the application. In other words, the interrupt handler
performs the operation correctly using the compressed
data format.
Data in compressed form is allowed to flow
to files and databases and so there is no need to capture
this I/O. Since 00 to 99 are maintained to represent the
years 1900 through 1999, there is no need to adjust or
compress constants used in existing programs.
If the program computes year values across
the year 2000 boundary, (e.g., 99 + 1) the remediation
technique is dependent on the machine language
instruction failing, for example with an overflow
interrupt, and on an interrupt hander simulating the
correct result (e.g., 99 + 1 = x'A0'). Note that this is
very dependent on the machine architecture, on the data
representation, and on the data format. It works on
System/390 packed decimal dates in Julian format (YYDDD),
but because of idiosyncrasies in the packed decimal data
format it does not work for Gregorian format (YYMMDD) or
with 2-digit years (YY). To handle Gregorian dates or
computations on 2-digit year fields, modifications to
source code are required in most cases. Note, however,
that these kinds of date computations which cross the
boundary do not occur in all programs and where they do
not, this remediation technique may not be restricted to
the Julian date format.
Obviously, dates need to be translated from
the compressed pseudo-hexadecimal format into 2 or
4-digit years as they flow from the program to screens
and reports. Tools that support this remediation
technique provide facilities to capture the data at these
points and to convert it appropriately. In some cases
this is not difficult to do; in other cases, it is very
difficult or not supported.
Note that this remediation technique will
only work in certain specific environments where the
applications use predictable and stylized data formats.
Not all program constructs are handled (e.g., tables
indexed by two digit years). It may require more
exhaustive testing of the remediated application than
source code remediation techniques. Also note, that like
any other remediation technique that uses compression, it
leaves stored data in a strange and unfamiliar compressed
format. Any application that touches the data may need to
be aware of and support this compressed format. This can
cause problems in the future and with systems that
interface to the remediated system.
In situations where this technique fits
well, however, the cost and time required for remediation
may be significantly reduced. In situations where this
tool is even only partially successful, it can be used as
a diagnostic aid in support of source code remediation.
For example, data on existing databases can be aged and
modified to this compressed format and the application
run with the supplied interrupt handlers set to record an
audit trail of all remediated instructions. This audit
trail can then be mapped back to the source code listing
to identify computations involving dates.
There are approaches other than the one
described above used by object code remediation tools on
the market that depend upon compression. In most cases we
are aware of, the techniques used by these other tools
are either very suspect or very similar to the techniques
in object code remediation tools that use windowing
(described below).
There are a number of remediation tools on
the market or soon to come to market, some of them
credible, and some of them suspect, that perform object
code remediation by tracing the execution of machine
language instructions in an object program and
identifying which instructions need to be remediated. The
process typically works as follows:
- using a variety of techniques, dates
that enter a program (from screens or databases)
are identified.
- using object code scanning tools or
other techniques the points at which I/O requests
are made in the program are identified and
breakpoints or "hooks" are inserted
into the object program.
- the program is then run under a
hypervisor that monitors the execution of
instructions and traces the flow of dates from
the identified I/O buffers as they flow through
the program. This process identifies other date
fields that may have been overlooked and dates
defined as constants in the object program (for
example, if a known date field is compared to an
unknown field, it is assumed that the unknown
field is also a date). The hypervisor may also
identify suspected dates it uncovers during
program execution -- data that is in a valid date
format and that contains values that could be
valid dates. Iterative testing under the
hypervisor may be necessary to confirm suspected
dates or to find other unidentified date fields.
- as the hypervisor identifies date
fields, it also identifies instructions that
process them and which require remediation. When
windowing is used for remediation, unequal
compares and arithmetic instructions that process
date data require remediation (note that this may
not be an exhaustive list).
- having found the instructions that need
remediation and having identified the formats of
dates they process, the tool provides an off-line
routine to patch "hooks" into the
instructions which require remediation. A set of
interrupt or "hook" handling routines
is also supplied.
- the patched object module is then run
in its normal environment (without the
hypervisor). As the hooked instructions are
executed, an exception handling routine gains
control and simulates execution of the hooked
instruction in the context of a defined fixed or
sliding window. Thus, for example, where the
original instruction might have performed a
2-digit year comparison incorrectly, the
exception handler will recognize that windowing
must be applied and perform it correctly. On
completion of its work the exception handler
returns control to the program and execution
proceeds.
This approach describes remediation using
windowing. A very similar approach could be implemented
using compression. In most situations, however, windowing
will be simpler, more successful, and will have fewer
undesirable side effects. This scenario describes one
reasonable, credible way in which this kind of object
code remediation can be done. Not all of the tools work
this way and not all are as credible or complete. Even
this scenario has holes (e.g., remediation of tables
indexed by years may require expansion of the table), and
may be very difficult to apply in certain common
situations (e.g., complex files or databases with variant
records). Nonetheless, the technique can work in certain
situations and may provide an alternative of last resort
where the source code for an important application is no
longer available. As with all object code remediation
techniques, this one should be used with caution, and the
remediated application should be extensively tested for
correct operation.
There are tools on the market that purport
to do object code remediation using expansion. The
algorithms used by these tools are, in some cases,
proprietary and confidential, and it is not clear how
effective they will be in general cases and on legacy
systems with procedural code. One can imagine that in
well designed object oriented environments where the
designer had the foresight to encapsulate dates as
objects, it might be possible to replace the date objects
and methods with ones that processed expanded dates. But
in procedural code, this is not simple. Date field
expansion often results in many side effects in
procedural code including buffer expansion and record
format changes. The necessary changes to object code
could be very extensive and difficult.
If this paper teaches anything, it is that
there is no silver bullet. Remediation is hard work, not
because the changes are difficult but because of the
number of cases that must be found and fixed and because
of the many variations in program constructs that must be
dealt with. Remediation is also difficult because testing
of systems changed to this extent is difficult, costly,
and exhausting. There is no getting around the testing
problem.
We will attempt here to provide some
comparisons of issues involved in the various remediation
techniques described above. This is not an exhaustive
comparison -- most of the details have been covered
above. Beginning with source code remediation techniques
we will focus on three areas: find/analysis, remediation,
and test.
The expansion technique has the advantage
that most of the find/analysis work is focused on finding
date field definitions that need to be remediated and not
procedural statements that must be changed. Proponents of
this approach overemphasize this point -- it is still
important to examine procedural statements to identify
literals that must be changed, tables that need to be
expanded or reorganized, headings that need to be
realigned, and algorithms that need to be changed.
The windowing technique requires that
procedural statements be remediated. In most situations
the process begins by finding date fields and then
finding cases in which they are used in procedural code.
Compiler assisted windowing has a real advantage here
insofar as the remediation is mostly done on data
definition statements rather than on procedural
statements, and the compiler can be used to identify most
procedural statements that require exceptional handling.
Compression requires a little of both --
data definitions must be found and changed, and
procedural statements must be changed in many cases to
accommodated unusual compression techniques (like
pseudo-hexadecimal).
Encapsulation requires dates to be found in
I/O buffers. This can be difficult in situations
involving complex databases and variant records. It also
requires certain modifications to be made in the
procedural code (literals, tables, algorithms, etc.).
Find/analysis tools that process source code
are common and mature. In most situations, perhaps with
the exception of encapsulation in complex systems where
find can be difficult, find/analysis issues should not
cast the deciding vote on the remediation technique to be
used.
Expansion and compiler assisted windowing
require fewer changes to the procedural sections of
programs than compression or other forms of windowing,
and they leave procedural code clearer and more
maintainable. For these reasons they are likely to
introduce fewer errors into applications during the
remediation process, and require less unit testing.
Expansion has the distinct disadvantage that
it requires changes to database, screen, and report
formats, and database content. All of this takes work, is
a source of error, and forces other changes in the
programs (e.g., realignment of headers on reports).
Compression implies changes to database
content, now or in the future, and compressed data
formats that can be a source of error, frustration, and
difficulty. While compression is not widely used, it is a
valuable technique for specific situations like dates in
database indexes. In such cases it is often used to
handle specific problems and another technique like
windowing is used for most of the rest of the work.
Encapsulation requires a set of carefully
architected changes to adjust dates as they flow in and
out of the program and non-intuitive changes to date
processing algorithms. The programmer must be taught to
think in the past and future simultaneously. While there
are common and mature tools that support other source
code remediation methods, there are few that support
encapsulation.
Firms that started Year 2000 remediation
some time ago typically used expansion, because it is
considered the best technique for the long term. Many
have found that expansion is a lot of work and that there
are not sufficient resources to complete all the work in
time. So, many of these firms and most firms that started
remediation later have turned to windowing as the
remediation technique of choice.
Most forms of windowing require a fair
amount of unit testing because a lot of changes are made
to procedural code. Expansion and compiler assisted
windowing result in fewer changes to the procedural logic
and thus less unit testing. Compression requires changes
to both data definitions and procedural logic, and the
testing of procedural logic may be more difficult than
with other remediation techniques because of the
complexities of the compression algorithm chosen.
Unit testing of applications remediated with
encapsulation is difficult and tricky. It needs to be
performed in the encapsulated environment, and it is not
always obvious which part of the code needs to be unit
tested. Typically with encapsulation, more focus is
placed on functional and system testing, but errors
encountered in these phases may be more difficult to
isolate and fix.
There is no substitute for functional and
system testing. It is hard and exhausting with any of the
remediation techniques. It is also more difficult to find
the source of errors discovered in functional or system
testing and so it is worthwhile to uncover and fix as
many errors as possible using unit test and code walk
through techniques.
Functional and system testing requires
access to aged data, and thus test databases must be
established. This is probably easiest with windowing
since database format and date data types do not need to
be changed. Test case and data distillation tools can
assist in this area.
Virtually everyone agrees that object code
remediation is more difficult technically and riskier
than source code remediation. It also has the
disadvantage that remediated applications are not easy to
change when maintenance is required. Nonetheless, in
certain cases object code remediation may be the only
feasible alternative short of rewriting the system from
scratch. And in others, where the application is not too
complex, and where there is a good match with the
abilities of the object code remediation tool, there may
be significant savings in the time and effort necessary
to complete remediation. While we have seen evidence of
this in individual situations, readers are warned that
this is not the general case and to be very careful in
evaluating the seductive arguments made by object code
remediation tool vendors.
Read this paper again. Most of the useful
information is in the detailed sections and not here.
There is no silver bullet. Watch out for
snake oil salesmen. Remediation is hard work. It is
typically harder to replace a home grown application than
it is to remediate it. If time and resources were
available, expansion is probably the best remediation
method. But in the real world, most enterprises are using
windowing. Compiler assisted windowing offers significant
advantages over other forms of windowing in situations
where the necessary compiler support is available.
Windowing is a good solution. But in the
long term you should be moving toward expansion. When
windowing is chosen, keep it simple. Choose a single
enterprise or department-wide window and deal with
exceptions as necessary. Supporting multiple windows in
the same application is hard and will be a source of
errors that are difficult to diagnose. Do not imbed
literal fixed window definitions in the code. Use a named
constant or a modifiable parameter to
define the pivot year and put its definition
into a copybook that can be included in many source
programs.
Object code remediation is hard and risky.
Use it only when there is a clear match or no other
alternative. And consider it a temporary solution.
Find/analysis and remediation are well
supported by tools in the market. Use them, and look to
reputable vendors to supply and support them. These
vendors need to be around after year 2000, because you
will probably need to continue to use the tools.
Testing is hard and exhausting work. Most
installations will not do enough of it. Be prepared for
ongoing problems after year 2000. Develop an automated
regression test bucket and use it on an ongoing basis
between now and year 2000 and after year 2000.
Establish clean management processes to
ensure that remediated applications are not incorrectly
modified by maintenance applied later. Where possible
choose a remediation technique (expansion or compiler
assisted windowing) which facilitates and enhances clean
management.
If you have the opportunity to establish
coding standards for the remediation project and for
future development, by all means do so. Choose a standard
for recording and processing dates -- use the ISO
standard of YYYYMMDD or YYYYDDD or the equivalent 2-digit
forms. Wherever possible use 4-digit years on APIs and
shared data that flow into or out of your organization.
If you do business with a government entity, you may be
forced to do this to meet bid requirements.
And by all means, plan to retire before the
year 10,000.