Pitfalls of OA Implementations
Date: 1995/11/27
Message-ID: <49d4uo$7as_at_hustle.rahul.net>
Hi There,
I have a high opinion of the skills and knowledge of some of the people who contribute to this group. I've written a paper which I've pasted below. I plan to polish it up and submit it for publication. All I've done so far is push it through a spell checker. If you have any comments, please send them my way.
I tried posting this to the oraapps list server, but it bounced since my rahul.net account does not subscribe.
-Dan
Daniel B. Bikle/Independent Oracle Consultant dbikle_at_alumni.caltech.edu | 415/941-6276 | P.O. BOX 70 LOS ALTOS CA 94023 http://www.rahul.net/dbikle
A Few Thoughts About Pitfalls of OA Implementations From a DBA Perspective Copyright Bikle Software 1995
A Few Acronyms
OA = Oracle Applications Software aka Oracle Financials/Manufacturing
RDBMS = Oracle RDBMS Software aka Oracle Server
DBS = An instance of Oracle RDBMS Software installed on a computer
aka Database aka Oracle Database
DBA = Database Administrator - Person who manages the RDBMS, each DBS
and some portions of OA
OS = Operating System aka UNIX or Windows NT or Novell
LAN = Local Area Network
SA = System Administrator - Person who manages the OS and LAN.
A Few Roles
End User = Person who uses OA functionality to implement business functions
like creating a GL journal entry or paying a vendor or creating a
purchase order or defining a part. He sometimes understands the
functionality of currently installed legacy systems.
End User Manager = Person who understands how business functions work in
concert to maximize some measurable result (profitability for example).
He allocates resources.
He defines security requirements in OA and sometimes the OS and DBS.
He sometimes understands the functionality of currently installed legacy
systems.
OA Consultant = Person who understands how business functions work in
concert to maximize some measurable result (profitability for example).
Person who understands how OA functionality maps to business functions.
Person who trains the End Users (and Managers) in OA functionality.
He sometimes understands the functionality of currently installed legacy
systems.
DBA = Database Administrator - Person who manages the RDBMS, each DBS
and some portions of OA. He understands interactions between the
RDBMS, the OS, OA, and the LAN.
SA = System Administrator - Person who manages the OS and LAN. He
understands interactions between the OS and LAN.
Programmer = Person who works with people in all of the above roles to make
enhancements to OA. Also he will create software which connects OA
to other systems.
Operator = Person who interacts with the various mechanical components of the
environment.
Typically the Operator feeds tapes to tape drives at night in concert
with backup software. Other tasks might include maintaining printers
and managing off-site tape storage.
Operations Manager = Person who ensures that mechanical components of the
environment have adequate power, space, fire control, and cooling.
Hardware Vendor = Company which provides the computer (HP or Sun for example).
SE = System Engineer. Employee of the the Hardware Vendor who works with
the Operations Manager, SA, and DBA to assemble the computer. Usually
he will install the initial version of the OS.
A Simplistic High Level Task List of an OA Implementation
The main purpose of this task list is to serve as a framework for the next section of the paper which addresses pitfalls. This task list is not a rigorous description of an OA Implementation. The author promotes himself as an experienced Oracle DBA, not a Project Manager or an OA Implementation expert.
Task A.
The End User Manager and an OA Consultant map business functions to OA
functionality to answer the simple question: Will OA allow us to run our
business?
Task B.
The End User Manager counts various objects within the company to answer the
simple question: How big of a DBS will we need?
Task C.
The End User Manager studies transaction rates within the company to answer the
simple question: How fast of a computer will we need to run the DBS?
Task D.
End User Manager decides if the following environments are needed:
TRAINING, STAGING, DEVELOPMENT, BACKUP.
-TRAINING is for training End Users
-STAGING is a duplicate of PRODUCTION. It is used for studying proposed
enhancements to the PRODUCTION environment.
-DEVELOPMENT is an environment for Programmers to develop enhancements.
-BACKUP is a duplicate of PRODUCTION.
Rather than having separate STAGING and BACKUP environments, some companies use the STAGING environment as a BACKUP environment should an emergency arise. The obvious risk to this approach is the STAGING environment might be in the middle of a test when the BACKUP environment is needed, but it is better than no BACKUP environment at all.
In addition, some companies prefer to avoid installing the TRAINING environment and instead send End Users off-site to OA classes.
Task E.
The DBA, SA, OA Consultant, and Operations Manager decide how the above
environments will get mapped to a set of computers. Usually TRAINING and
DEVELOPMENT share a "small" computer. PRODUCTION, STAGING, and BACKUP
usually do not share a computer. Usually PRODUCTION gets a faster more
expensive machine. Often the BACKUP environment is several hundred miles
away from PRODUCTION as might be specified in a disaster recovery plan.
Task G.
Once the team members (DBA, SA, OA Consultant, and Operations Manager) decide
how the environments will be distributed among a set of computers, they
purchase the computers from the Hardware Vendor along with an OS license
consistent with the number of concurrent users who will be logging into each
computer. The computers and OS are then installed by the SE. Next, the SA,
Operations Manager, and Operator work together to ensure that backup and
restore procedures work reliably.
Task H.
While waiting for the computers to arrive, the team members purchase a set of
Oracle licenses consistent with the environments defined beforehand.
Task I.
Next, the DBA installs the TRAINING environment and shows the OA Consultant
how to access it. The OA Consultant begins training the End Users, and End
User Managers.
Task J.
Then, the DBA installs the PRODUCTION, STAGING and BACKUP environments. Part
of this installation involves the creation of a scripts which copy PRODUCTION
(data and software) into the STAGING and BACKUP environments.
Task K.
Next, the DBA installs ("define" might be a better verb) a mechanism which
copies enhancements which have successfully passed testing in the STAGING
environment into the PRODUCTION environment. This is a complex topic which
will be mostly ignored in this paper because entire books could be devoted to
the subject.
Task L.
Then, the DBA installs the DEVELOPMENT environment. Part of this environment
should contain a version control system which allows developers to easily
trace dependencies between files and database objects.
Task M.
Next, the DBA and SA install OA client software if the company decides that
its End Users should access OA data from client software running on PCs.
This task requires the software to be installed on a file server and then
mapped to a network drive on each PC. If the team decides to install several
OA server environments (TRAINING, PRODUCTION, STAGING, BACKUP, and
DEVELOPMENT), the team would also specify a need for several OA client
environments.
Task N.
Then, the DBA, OA Consultant, Programmer, End Users, End User Managers work
together to copy data from the legacy system into the OA PRODUCTION
environment. Typically this team will decide on which sets of data will be
copied while they complete Task A. and Task B.
A Brief List of Pitfalls
Task A.
This task is so enormous it could have no ending. One approach taken by an
OA Consultant was to create an environment called CRP(Conference Room Pilot).
He worked many weeks with the customer to fill CRP with "real data". Next,
they walked through a great variety of business functions. Simple functions
like enter an order, define a customer, issue a 1099. Then they attempted
more complex functions like "Month-end Close" and MRP.
Task B.
This is another time consuming task. Also it will lead to battles between
the OA Consultant and the End Users over historical data. If historical data
is left on the legacy system rather than mapped and then copied into the OA
system, the OA Consultant will have less to worry about. The End Users will
dislike switching between two systems during their work days. If later
(during the implementation of Task N.) insufficient data is copied into OA
from the legacy system, End Users will be forced into "data entry grunt work"
to complete some of their business functions. For example, suppose an End
User needs to enter an order into the OA Order Entry module from an existing
customer. This function would be easier for him to implement if data about
the existing customer already existed in the OA DBS (meaning that the OA
Consultant had done a good job of completing Tasks A, B, N.).
Task C.
This is another daunting task. The only rigorous way to complete this task
is to create a CRP environment which is a duplicate of the proposed
PRODUCTION environment (which is probably not well defined yet). Next, the
DBA and OA consultant work together to run End User emulation software
against the CRP environment. This approach has been taken several times by a
large computer manufacturer in Mountain View. It works well in an
environment where the client side of the the CRP environment is on UNIX.
Correct completion of this task would allow them to say something like this,
"We can enter 20,000 GL journal entries into the OA GL module where the DBS
is running on an XYZ server with 1 Gb of ram, and 32 2Gb disk drives on 8
fast wide SCSI controllers..." A less rigorous way to complete this task is
to find other OA customers and ask them what kind of transaction rates they
support on their PRODUCTION environments. The following groups of people
contain a high density of OA customers:
Usenet Group: comp.databases.oracle
User Group: Oracle Applications User Group (OAUG)
User Group: Northern California Oracle User Group (NCOUG)
E-mail List Server: oraapps-l_at_cpa.qc.ca (subscribe via listproc_at_cpa.qc.ca)
Finding out the latest information about how to join the User Groups is best done by posting a question to comp.databases.oracle.
Task D.
This task is easy. Once the decision has been made, it is important that
everyone be trained on why we have the different environments and how to
access them once they are available. Actually this is only possible in a
small company. The OA Consultant is responsible for this education effort.
One pitfall customers keep falling into is a community of End Users who lack configuration information. So sometimes they make mistakes like logging into the TRAINING environment when they are supposed to be logging into the PRODUCTION environment. This problem will become more acute with the advent of the OA PC client software. The End Users need to know which sets of client software on the file server should be used with each OA environment. Since it is inevitable that End Users will become confused about the different environments, they need to be given a single point of contact, preferably the OA Consultant, to obtain configuration information. Otherwise they might start asking inappropriate people (like the SA or DBA) where they should be "pointed at". Quite often this problem will get tangled up with network configuration problems on the End Users desk top environment (usually a PC). One scenario might be this: The End User knows he wants to be pointed at PRODUCTION but he sees that he is pointed at TRAINING. He calls the SA who lacks the training and/or knowledge to point the PC at PRODUCTION. All he knows is IP addresses and port numbers. He doesn't know how different nodes on the network are being used. Someone in the organization needs to know enough about the high level business goals and the low level network configuration to solve this kind of problem. Usually that person is the OA Consultant or the DBA.
Task E.
This task is easy since financial constraints usually limit how many
computers a company can buy. From purist perspective, a company would supply
one computer for each environment. From an administrative perspective, it
might be tempting to place all of the environments on a single computer.
If the team decides to place PRODUCTION and STAGING on different computers, it would be ideal if the two computers were exact duplicates. One corner which could be cut is a lower number of processors and slower processors (but they should remain the same architecture [Sparc, PA Risc, Intel, ... ]) on the STAGING machine. The main consideration when specifying the computers for TRAINING, DEVELOPMENT, and BACKUP is attaching adequate disk space to the machines and keeping them all on the same OS and hardware architecture.
Task G.
Two large pitfalls await us when we implement this task. First, many weeks
will pass between the time the hardware order is placed in the Vendor's order
entry system and the day that the computers and peripherals start arriving on
the loading dock. Most major computer manufacturers have a backlog of
orders. Also if the computer is large enough, the manufacturer will assemble
it on a shop floor and test that it works before they ship it, which of
course takes time. Second, once the hardware order does arrive, parts of it
will not match what the customer is expecting. Many other pitfalls are
associated with this task:
-Placing the order is extremely tedious due to the modularity and complexity of
large UNIX machines.
-Placing the order is tedious due to discounts available to the customer because
of contractual relationships between the customer and the vendor. For example
some customers will become a VAR just so they can get these discounts.
-SEs are often not trained well enough to flawlessly assemble these large systems
once they do arrive on the loading dock.
-The paper work (aka manifests) which comes with the hardware is usually both
cryptic and voluminous.
-Operations Managers are sometimes given inadequate information before the
systems arrive; when they do arrive sometimes some resource is lacking to
support them (usually power, cooling, and space).
One "Ace in the hole" the customer might play if the ordering process goes wrong, is to switch hardware vendors. The big vendors in the UNIX marketplace are listed below:
Sun, HP, SGI, Pyramid, Sequent, IBM, DG, DEC, and AT&T.
Using this technique assumes of course that the team ordering the hardware has the power to switch hardware vendors. Often this is not the case since the decision to buy large systems from a specific vendor comes from the CIO, or CFO, or CEO. Obviously, a CIO with perfect information would consider the reliability of a hardware vendor's whole ordering apparatus before issuing an edict that "We only buy our computers from XYZ, Inc."
Task H.
In some ways, this task has the same pitfalls as Task G. One difference is
the customer cannot threaten to buy OA from another vendor if Oracle is doing
a poor job of handling the software order. One pleasant trait that Oracle has
is that it usually ships more software products than the customer has
ordered. Oracle relies on the customer to use only the software which he has
a license for. This practice generates goodwill and reduces administrative
headaches. The downside, of course, is that some user at the customer site
might start using some software which is unlicensed. The customer would then
be obligated to buy a license for that software module.
Task I.
This is a simple task. Oracle provides a ready made demo environment which is
suitable for training people who need to interact with OA.
Task J.
This is a simple task. The scripts which copy PRODUCTION into STAGING would
be divided into 2 types. The first type would rely on the Oracle
EXPORT/IMPORT utilities to copy the data. The second type would rely on a
UNIX utility to copy the files (tar or cpio perhaps).
The scripts which copy PRODUCTION into the BACKUP environment might need to be more sophisticated depending on how closely BACKUP needs to mirror PRODUCTION. Some of the later versions of the RDBMS contain replication features which could be used to keep the BACKUP DBS in synchronization with PRODUCTION on a transaction by transaction level. The obvious problem with this approach is a degradation of performance.
Task K.
Enhancements come from Oracle on tape or they come from on-site developers.
Perhaps the most common pitfall is the DBA loosing track of which
enhancements have been installed on which environments. Another pitfall is
the End Users not adequately testing enhancements on the STAGING environment
before the enhancement is installed on the PRODUCTION environment. Another
problem is the size of the enhancements. For example, a customer that wants
to upgrade its PRODUCTION environment from OA version 9.x to 10.5 might need
to take the PRODUCTION environment off-line for several days during the
upgrade.
Task L.
Installation of the DEVELOPMENT environment is an easy task. Developing
enhancements to OA is complex. In fact, whole text books have been written
about the subject of Software Engineering. With that in mind we will jump
in, gloss over the subject, and say this:
-Many dependencies exist between software modules and database objects.
-These dependencies are subject to change whenever Oracle releases a
new version of OA.
-The dependencies created by Oracle Corporation are not always documented
accurately or thoroughly in the "database design" documents.
-The design philosophies behind many of the database objects and software
modules are not stated anywhere.
-The dependencies and design philosophies created by the on-site developers are
tedious to document.
Assuming that the programmers have a good system for managing their source code and database objects, they must tackle the problem of eventually placing their bug-free enhancements on the PRODUCTION environment without disrupting business functions. Ideally they would follow a systematic procedure that could be modeled by a flow-chart:
Develop enhancement
Document enhancement
|
V
Develop installation script
Document installation script
|
V
Copy PRODUCTION to STAGING
|
V
Install enhancement in STAGING
Test enhancement
Make sure it works
Make sure it does not break anything else
|
V
Install enhancement in PRODUCTION
Test enhancement
Make sure it works
Make sure it does not break anything else
So we can sum all of this up into a single pitfall: Creating enhancements to a PRODUCTION environment is a complex task which requires rigorous controls to prevent errors from corrupting data.
Task M.
This is an easy task if the team is well organized. One problem with PCs
which can turn this task into a pitfall is the SA cannot work on a PC from a
remote site. He must be sitting at the PC if he needs to change its
configuration. This can be time consuming and inconvenient for both the SA
and the PC owner. Since it is a manual process, it cannot be automated like
it can for a network of UNIX clients.
If an End User needs to constantly toggle between the TRAINING environment and PRODUCTION environment, the SA should setup some kind of mechanism to make it easy. Another problem with PCs is they sometimes work very poorly in a networked environment compared to UNIX machines. A PC that cannot work correctly as a network node might add a significant number of hours the SA workload. Another problem with PCs is they sometimes abnormally disconnect from the network which causes problems (Locks and Latches not going away) with the DBS because "it thinks" the PC is still attached.
Task N.
This is one of the most complex tasks of the entire implementation. The
major pitfall is the long length of time the team needs to map data elements
in the legacy system to either interface tables in OA or the actual data
tables. Also a great deal of time will be needed to build scripts (once
their specifications are written in the previous step) which copy the data
out of legacy system into appropriate OA tables. The way some companies deal
with this long length of time problem is they implement OA in phases.
Unfortunately this sometimes means that the legacy system will need data from
OA while the company is in this state of flux. A good example of this is
data about customers. Each system needs up-to-date customer data. Once the
OA system is at a point where it can start using its own customer data, the
company might decide that all new data about customers should be entered into
OA and then copied to the legacy system where it is needed to keep some other
business functions running there until they can be converted to the OA
system. Creating a mechanism to copy this data is a bad side effect of
complexity and is a classic example of what business process modelers call a
positive feedback loop.
Daniel B. Bikle/Independent Oracle Consultant dbikle_at_alumni.caltech.edu | 415/941-6276 | P.O. BOX 70 LOS ALTOS CA 94023 http://www.rahul.net/dbikle
Received on Mon Nov 27 1995 - 00:00:00 CET