RE: [Fwd: UNIX Performance Issues]

Hi Rahul.  

Interesting, as ever!
See below  

James

--
James Morle
Scale Abilities, Ltd
http://www.scaleabilities.co.uk <http://www.scaleabilities.co.uk/> 
Author of "Scaling Oracle8i - Building Highly Scalable OLTP System
Architectures" 

-----Original Message-----
Dandekar
Sent: 19 February 2002 15:49
To: Multiple recipients of list ORACLE-L


James,
 
Getting interesting, isn't it? I have added my response...
 
----- Original Message ----- 

To: Multiple recipients of list ORACLE-L <mailto:ORACLE-L_at_fatcity.com>  
Sent: Tuesday, February 19, 2002 8:58 AM


Rahul,
 
Here's what I would do. 
1) I would use "mpstat" for the processor statistics. This breaks the
usage up by processor in SMP configurations. This can be useful to see
the relative loading of each CPU, in particular the breakdown of kernel
and user time.
2) Memory: Concentrate on Page Outs and Free Memory more than anything
else. That will give you plenty of clues about memory starvation, and
the relevence of your VM tuning.
3) I/O: User "sar -d". It's a bit annoying on a system with a lot of
disks, because it returns a row for every device, even if no I/O
occurred in the sample period. However, it makes it easier to parse. ;-)
Notably, keep an eye on the Service Times (avserv?), Wait times


(avwait), and the queue depth. The utilisation is a function of these


(queuing theory), but you can store that too as a shortcut. You can give

sar any sample period, so your 5 minute averages are no problem.

How can I get the current I/O load on the system? I don't know exactly
what metric I am looking for.
But I want to establish some baseline metric for each machine and then
hunt for spikes from the
gathered data. The metric can be "I/O load on system bus in Mb/sec"


(like the netstat info packets


input and output). I don't want individual disk statistics. I just want
a overall number, which I
can snapshot. 
 
I know what you're after, but it's just not going to work that way! A
network adapter is a single serial resource with a finite limit. An I/O
subsystem is an arbitrarily complex *set* of resources with a finite
capacity on each! For example, if you were to just measure the aggregate
I/O rate across your SAN (or whatever), that may well return a good
number. However, one disk in there could be assuming 50% or more of the
load due to hotspots. This disk would probably be providing multi-SECOND
response time, and because it's the hot disk, will be slowing nearly
everything down. Your aggregate stats would not show this. You need
per-disk, per-controller, and if you've got a very busy system you might
want to start worrying about backplane capacity. There's no easy way to
measure that one, however.

4) Network: "netstat 5" will report a row for every 5 seconds (for
example), showing how many packets went in and out of each interface.
Your question below is easily answered - you have two columns in your
output; the first is for the named interface (hme0), the 100baseT
network card. The second is a total of all cards - looks like you only
have one. This total can also include the loopback interface (lo0), so
look out for that.


If I have only one card then why the total and hme0 data are different


(by about 10%)? 


 
I suspect it is reporting the lo0 interface in the total, but not
showing it individually. Check out the options for netstat (I don't have
Slowlaris in front of me right now).

Good luck, you're doing the right thing. I've been working on some
software to do just this for a couple of years. I'd love to hear how it
goes!

+Rahul

Regards
 
James
--
James Morle
Scale Abilities, Ltd
http://www.scaleabilities.co.uk <http://www.scaleabilities.co.uk/> 
Author of "Scaling Oracle8i - Building Highly Scalable OLTP System
Architectures" 

-----Original Message-----
Dandekar
Sent: 19 February 2002 12:59
To: Multiple recipients of list ORACLE-L


James,
 
Interleaved, please find my reply....
 
+Rahul

----- Original Message ----- 
To: Multiple recipients of list  <mailto:ORACLE-L_at_fatcity.com> ORACLE-L 
Sent: Tuesday, February 19, 2002 6:03 AM


Rahul,
 
Did you get a response on this? I'm not sure I fully understand the
actual question - are you looking for specific commands you need to run
to get the information, 

[Rahul] Yes. I would like to know which flags of the commonly used
commands give good information.
For general System stats, I use "sar -u" (same as default), for Memory /
Virtual Memory I use "vmstat"
and look for "r   b   w   swap   free   pi   po   us   sy   id" columns.
I am looking for general monitoring. And once we have this general
information giving a overall picture,
we could know if there is a problem and we could investigate further.
I am specifically looking for IO and Network statistics.
Is there any command which would give me approx IO of the system, say in
last 5 minutes or
current?
How to get network statistics? I was littlebit confused with netstat.
There are two main categories
in my output : hme0 and Total. What does that mean?
    input   hme0      output           input  (Total)    output
packets errs  packets errs  colls  packets errs  packets errs  colls
5757291 0     2447690 0     0      6071152 0     2761551 0     0
45      0     1       0     0      45      0     1       0     0
24      0     2       0     0      24      0     2       0     0
 
What I plan to do is to take snapshot of all these statistics at a
certain frequency and put it
in database. Later on I could generate reports based on this.
Currently, I have a lot of "Camera"s like this taking snapshots of my
system.
Others involve Oracle stuff like DB Size Growth, Performance Ratios,
UNIX File System
usage, Replication Statistics, Growth of DB objects, a lot of monitors
for application
info (e.g. total # of clients, # of invoices generated per day).
I generate trends based on this archival data for capacity planning and
proactively
anticipating chronic problems.

or advice on how to interpret it? Don't forget that you will really need
to correlate many of these 
statistics to the Oracle pathology at the same time. 

You said it! I want co-relation of Application Load, UNIX System Load
and Database Statistics.
And not just when the problem arises. So, that's what I am trying to
develop.
 

This then causes a problem because your sample points will at the very
least experience clock drift and become harder to compare over time.
There are ways to solve it, though. 
Anyway, if you could elaborate a little, I can try to assist!
Regards
 
James

--
James Morle
Scale Abilities, Ltd
http://www.scaleabilities.co.uk <http://www.scaleabilities.co.uk/> 
Author of "Scaling Oracle8i - Building Highly Scalable OLTP System
Architectures" 

-----Original Message-----
Sent: 18 February 2002 22:11
To: James Morle


Hi James,

I've got no idea whether this is of interest or not to you, but you
probably know a bit about this topic.

Mogens

-------- Original Message -------- 
Date: 	Thu, 14 Feb 2002 07:43:26 -0800	
<rvd_oracle_at_hotmail.com>	
Reply-To: 	ORACLE-L_at_fatcity.com	
Organization: 	Fat City Network Services, San Diego, California	
To: 	Multiple recipients of list ORACLE-L
<mailto:ORACLE-L_at_fatcity.com> <ORACLE-L_at_fatcity.com>	


DBAs,



This might be littlebit (or completely!) UNIX related... But I am told

to do the performance analysis of some 10-15 machines and generate

some statistical data to find out bottlenecks and identify areas of

tuning...



Operating System : Solaris 2.6



I have been using sar, iostat, top...

I actually plan to script these things and run these scripts at certain

intervals and put the data in database (Oracle 8i) and then do the

crunching...

Inputs are appreciated...



1. I/O

   What is current I/O status. Is there a lot of I/O going on?



2. Paging

   Is there lot of swapping / paging happening?

   Which processes are getting swapped in/out continuously?

   Are the I/O waits due to swapping / paging or regular stuff

   like DB waiting to read from DB files?



3. CPU

   What is the CPU utulization? Which processes are using lot of CPU?



4. Memory

   What is the current picture of Real and Virtual Memory?

   What processes are using how much memory? Which processes

   are i

n real memory and which are in virtual memory?

   Which processes are swapped in and out from/to real/virtual memory

   and how many times?



5. Network

   What is the percentage utilization of network pipe?

   What is the capacity (bandwidth) of the network device?

   What percentage of that bandwidth is getting used?

   Is the system waiting for data from outside network I/O?

   In short, is there any bandwidth problem with network device

   or network traffic.



Thanks,



      ___           ___        ___           ___           ___

     /  /\         /  /\      /  /\         /  /\         /  /\

    /  /::\       /  /::\    /  /:/        /  /:/        /  /:/

   /  /:/\:\     /  /:/:|   /  /:/        /  /:/        /  /:/

  /  /::\ \:\   /  /:/|:|  /  /::\  __   /  /:/  ___   /  /:/

 /__/:/\:\_\:\ /__/::\|:| /__/:/\:\/ /\ /__/:/  /  /\ /__/:/

 \__\/~|::\/:/ \__\/\:\:| \__\/  \:\/:/ \  \:\ /  /:/ \  \:\

    |  |:|::/     \__\::|      \__\::/   \  \:\  /:/   \  \:\

    |  |:|\/ 

      |  |:|      /  /:/     \  \:\/:/     \  \:\

    |__|:|         |__|:|     /__/:/       \  \::/       \  \:\

     \__\|          \__\|     \__\/         \__\/         \__\/



-- 

Please see the official ORACLE-L FAQ: http://www.orafaq.com

-- 

Author: Rahul Dandekar

  INET: rvd_oracle_at_hotmail.com



Fat City Network Services    -- (858) 538-5051  FAX: (858) 538-5051

San Diego, California        -- Public Internet access / Mailing Lists

--------------------------------------------------------------------

To REMOVE yourself from this mailing list, send an E-Mail message

to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in

the message BODY, include a line containing: UNSUB ORACLE-L



(or the name of mailing list you want to be removed from).  You may





also send the HELP command for other information (like subscribing).








-- 
Please see the official ORACLE-L FAQ: http://www.orafaq.com
-- 
Author: James Morle
  INET: James.Morle_at_scaleabilities.co.uk

Fat City Network Services    -- (858) 538-5051  FAX: (858) 538-5051
San Diego, California        -- Public Internet access / Mailing Lists
--------------------------------------------------------------------
To REMOVE yourself from this mailing list, send an E-Mail message
to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
the message BODY, include a line containing: UNSUB ORACLE-L


(or the name of mailing list you want to be removed from).  You may


also send the HELP command for other information (like subscribing).