Brent Martin

Lately at work I've been having to remind myself lately not to pave the cow path -- not to bring existing business processes and technology forward just because that's how we've always done it.  Here's the poem that the expression comes from.

One day, through the primeval wood,
A calf walked home, as good calves should;
But made a trail all bent askew,
A crooked trail, as all calves do.

Since then three hundred years have fled,
And, I infer, the calf is dead.
But still he left behind his trail,
And thereby hangs my moral tale.

The trail was taken up next day
By a lone dog that passed that way;
And then a wise bellwether sheep
Pursued the trail o’er vale and steep,
And drew the flock behind him, too,
As good bellwethers always do.

And from that day, o’er hill and glade,
Through those old woods a path was made,
And many men wound in and out,
And dodged and turned and bent about,
And uttered words of righteous wrath
Because ’twas such a crooked path;
But still they followed — do not laugh —
The first migrations of that calf,
And through this winding wood-way stalked
Because he wobbled when he walked.

This forest path became a lane,
That bent, and turned, and turned again.
This crooked lane became a road,
Where many a poor horse with his load
Toiled on beneath the burning sun,
And traveled some three miles in one.
And thus a century and a half
They trod the footsteps of that calf.

The years passed on in swiftness fleet.
The road became a village street,
And this, before men were aware,
A city’s crowded thoroughfare,
And soon the central street was this
Of a renowned metropolis;
And men two centuries and a half
Trod in the footsteps of that calf.

Each day a hundred thousand rout
Followed that zigzag calf about,
And o’er his crooked journey went
The traffic of a continent.
A hundred thousand men were led
By one calf near three centuries dead.
They follow still his crooked way,
And lose one hundred years a day,
For thus such reverence is lent
To well-established precedent.

A moral lesson this might teach
Were I ordained and called to preach;
For men are prone to go it blind
Along the calf-paths of the mind,
And work away from sun to sun
To do what other men have done.
They follow in the beaten track,
And out and in, and forth and back,
And still their devious course pursue,
To keep the path that others do.

They keep the path a sacred groove,
Along which all their lives they move;
But how the wise old wood-gods laugh,
Who saw the first primeval calf!
Ah, many things this tale might teach —
But I am not ordained to preach.

Oracle announced today at the Alliance '13 conference that the PeopleSoft 9.2 will be generally available on March 22.

Here's the link to the press release:  http://www.oracle.com/us/corporate/press/1920557

So far I've written about how you might approach the plan, design, build and test phases of a PeopleSoft replatforming project.  This time around I'd like to spend some time on the Cutover.

You’ll probably want to do at least 4 mock cutovers. One to build the initial development environment on the new hardware.  One to start System Test. One to start User Acceptance Testing, and a “dress rehearsal” to prove out your cutover plan/strategy. 

Start the cutover plan when you do your first migration. Capture tasks and timing. And continue to refine it with each additional mock cutover.

For the 3rd mock cutover, include items in the cutover plan for communication, external systems that will need to be modified and moved in parallel, shutdown sequence for batch, expected timeline, contact lists, etc.  By now your communication plan should be fairly explicit and there should be no surprises from the extended IT team or the business as to what will happen and when.

One to two weeks prior to cutover, execute a “dress rehearsal” where you actually move your production database in as realistic of a fashion as possible.  Validate your final timings and make sure nothing was missed.

 Two words about cutover communications:  They’re important.  You need to keep all of your stakeholders informed of where you are in the cutover, raise any issues quickly, and insure all of the hand offs are executed cleanly with no loss of time.  Identify a single point of contact (or contacts if you’ll be running cutover around the clock) who can get status from the team members without bugging them too much and prepare regular communications to the interested stakeholders.   

 In addition, you’ll probably want to maintain two open conference call bridge lines:  One for executive/stakeholder updates, and another to allow your technical teams to quickly collaborate on handoffs or issues that arise.

 A good cutover plan will include a final “Go/No-Go” decision point prior to starting any cutover activities.  If you have no “Severity 1” or “Showstopper” issues the cutover should proceed on schedule.

 Now the cutover plan becomes the script for everything over the next hours and days.  A common scenario follows:  Users close out transactions.  Batch schedule is stopped in a controlled manner. Final interface files are sent.  Validation reports are run that users will use to validate the system when it comes back up.  Finally user accounts are disabled, the application is stopped, and the DBA team (who is hopefully caught up on sleep) takes over.

 Now the DBA team executes the data migration using whatever tool you decided on.  Row count reports and other validation will be executed when it’s complete and the PeopleTools upgrade will start on the database.  This can be the longest part of the process.  Then all of your customizations are migrated in, the application is configured and a non-destructive technical checkout is conducted.

 It’s typical at this point to allow a limited number of users log in and enter and process real production transactions. This allows any problems to be identified and resolved before the system is turned over to the larger user population.

 Finally we’re ready to go.  Get your project sponsors and executives on the phone for a final Go/No-Go decision.   Once you get the green light, unlock all of the users and start your batch schedule back up in a controlled manner.  Congratulations!  This is a big accomplishment!!

So far I've written about how you might approach the plan, design, and build phases of a PeopleSoft replatforming project.  This time around I'd like to spend some time on the Test phase.

Just because you’re only changing some SQL around doesn’t mean that you can take shortcuts with testing.  You’ll want to run an entire stem-to-stern test of your PeopleSoft system to insure that everything works as expected.

 One thing to keep in mind:  99% of your defects will be with custom and customized code.   The delivered code won’t generate nearly as many problems, so if you’re deciding where to spend your testing resources definitely focus on the custom processes.

 As I mentioned in the Build phase, Unit Testing is critical.  It’s arguably the most important testing that you will do so come up with a mechanism to track unit test results with the objects modified and make sure you have 100% custom code coverage.

 Testing the data migration process is important too.  Databases differ in subtle ways and you’ll want to make sure your normal and extended character sets make it across correctly.  Naturally you’ll want to run row count reports, but you’ll need to go beyond that.   You’ll want to make sure the data in the underlying tables are identical and nothing has been lost in translation.  One simple way is to use ODBC to join to tables in the source and target databases and create queries that insure the individual columns are identical.  Unfortunately that approach is extremely slow.  Another approach is to run hash algorithms on key character fields and summarize the totals, comparing the results on both the source and target database.

System/Integration Testing is also very important.  For one thing, you’ll want to have confidence that the system behaves the way you expect it to.  And interfaces will generate problems themselves.  One common problem is that your interface programs probably assume the default date format for a database platform, and interfaces that don’t specify a date format can choke when the incoming date format doesn’t match what’s expected, or they can send the wrong date format to an output file.  Switching from a Non-Unicode database platform to Unicode can cause other problems.  You’ll want to execute all of your interfaces and make sure results are valid. 

User Acceptance Testing is important as well.  Users who know the processes should spend some time in the system making sure all of the critical functionality works and they feel comfortable everything is working as expected. They’ll need to be on the lookout for date format issues, performance issues, data entry discrepancies, etc.  They should also spend quality time with their reporting to make sure no new errors were introduced during the build phase.

And finally Performance Testing should be conducted to flesh out any new problems introduced by the new platform.  The performance testing should include Online Performance testing,  Batch performance testing, and if possible data from PeopleSoft Performance Monitor should be captured and compared to baseline performance data from the old system. 

Online performance testing is typically conducted using a tool like LoadRunner or Rational Performance Tester.  You record scripts based on a set of highly used business processes and play them back in volume.  While the test is executing your admin team will monitor system performance on the various servers and look for bottlenecks.  At the end of the day this is necessary for a performance testing effort (especially if you’re making multiple changes like migrating to new hardware and/or a PeopleTools release).  However, it’s not sufficient to identify all performance issues.

One of the big deficiencies of online performance testing is that it doesn’t test batch jobs, or if it does test them it is very limited.  For batch testing, you’ll want to define a testing mechanism that is realistic according to real, observed jobs that run in your production environment.  You’ll want to make sure that the jobs have real data to process.  And you’ll want to make sure the jobs are sequenced in such a way that dependencies are tracked to some extent.  After all of that, you’ll want to come up with a way to track and report results.  I’ll write more about the approach and toolset I’ve used to get realistic batch tests in the future so stay tuned. 

The other deficiency of online performance testing is that PeopleSoft is just too complicated of an application to expect a robot to crawl all of the web pages looking for slowness.   Sure, during yoRead More...

In last two postings I wrote about how you might plan a project where you migrate your PeopleSoft application from one database platform to another, and how you might approach the Design phase.  I wanted to share my thoughts about the Build phase in this article.  I'll share my thoughts about the Test and Cutover phases in my next posting(s).

The Build Phase

The Build Phase is always my favorite part of any PeopleSoft project, probably because I come from a Development background.  The Build phase of a replatforming project is in some ways very straightforward, and in some ways it is more difficult.  The problem isn’t in the coding changes – it’s not too difficult to make a piece of SQL work on a different database platform -- the challenge is in Unit Testing.  Every SQL that is touched must be unit tested, and that will be the biggest part of the effort.

Most developers are used to unit testing their own work.  But it's a good idea to use a code and testing review where developers document each object change and unit test and another developer reviews the results.  Since there will be many small changes, the documentation requirement should be light, but it should include a trace file that proves that each App Engine step, PeopleCode SQL, and SQR Function was executed was tested.  How structured your process is will depend on the size and location of your team.  Insuring quality with process and documentation might not be as important in a small shop, but is critical to your success if you have a large development team located off shore.

Unit testing is the only opportunity you’ll have to actually test each piece of modified code.  Subsequent phases will test the system overall, but you will probably not achieve 100% code coverage.  Fortunately, almost all of your defects can actually be caught in unit testing of a replatforming project so you should use this to your advantage.  Defects that get missed will haunt you in later testing phases where they’ll be more visible and more expensive to fix.

Also as part of this phase, your DBA team should  execute another mock cutover using the tools and steps you decided you will use for the real cutover.  The resulting database (plus the code generated in the Build phase) will be the starting point for your first test database.

And the testing team should start building the test scripts for the subsequent test phases here.  Since we’re not changing application functionality, they should be able to leverage existing scripts from prior upgrades or implementation and enhance them for functionality that was added since the original scripts were created.