RE: Interview Questions for a Unix Solaris System Admin

Well, the purpose of the original post was to list a set of questions that, while admittedly "academic" in that they involve facts, cross 1) a broad cross-section of what a sysadmin might be asked to do and 2) range from easy to difficult.

The point is not to know the answer to every question, though if they do, bully for them. The point is for what they don't know, can they use related knowledge they do have to answer the question? I was always far happier when someone said something like, "You know, I'm not sure what isn't stored in an inode, but because symbolic links can have their own modification time, I'm going to guess its the modification time" rather than "I don't know". That is a wrong answer, but the fact that he was able to logic out what is a reasonable conclusion, is the sort of thing I like in people that work for me.

It's true, some of the questions are esoterica, but again, its how they answer the question. There's a post I sent that has yet to arrive (in my box anyway) where I answer the questions. For the question about full-duplex, I can't remember the flags you set either - and I'd be fine with anyone who said, "You know, I'd man the ge driver and then use ndd to set what I had to". The point is they knew to look up the ge driver and use ndd.

Further discussion inline....

> > 1) What is an inode? Bonus: What important piece of file
> > information is NOT stored in the inode?
>
> Might be better to ask the pros and cons and having many
> inodes as opposed to few; circumstances when you would not
> want to go with the default number you get when you create a
> file system (never happened when I was a sys admin).
>

<Very pertinent info snipped>
> You might ask the circumstances under which the admin would
> AND has used soft links. In the past, I had the unpleasant
> experience of inheriting servers where the previous admin(s)
> had made extensive use of soft links resulting in boxes rife
> with spaghetti file systems. This usually happens because,
> for one reason or another, a bunch of files need to be moved
> from one place to another; then instead of fixing whatever
> uses those files to tell it to go the new location, a soft
> link to the new location gets created as a quick and dirty
> solution. These things accumulate, and eventually you have a
> mess and booby traps ready to blow up.
>

These are equally valid questions, but I have found that by and large, sysadmins might know from reading a book somewhere that you need one inode per file and that hard links and sym links are different, but knowing what an inode is indicates that not only are they aware of all of the above but they can in-depth explain what is happening to the system.

> > 2) What is priority paging and how does it work? (mildly dated, but
> > useful if they claim to have been around for a while)
>
> Never heard of it. Sounds like a term for a type of swapping
> (lazy versus not lazy).

Solaris-ism with 2.6 (the original question was for a Solaris admin)

> > 3) What does sr stand for in vmstat output?
>
> man vmstat.
>
> > vmstat
> Virtual Memory Statistics: (pagesize = 8192)
> procs memory pages
> intr cpu
> r w u act free wire fault cow zero react pin pout
> in sy cs us sy
> id
> 10 1K 49 914K 279K 90K 130G 500M 1G 363 615M 2
> 389 44K 3K 17 13
> 70
>
> Ummm ..... got no sr here ....
>

Apparently you're not on a Solaris system - count yourself lucky. ;)  

> > 4) How would I configure the gigabit ethernet interface to
> force it to
> > be full duplex?
>
> I guess if this were a common daily activity, you would want
> somebody to know how to do this. For most places, you might
> do this ... ohhhh ... maybe once per year ... maybe.
>

I concur - but knowing that you use ndd to get/set network device driver parameters is the real thrust of the question. Asking about gigabit specifically gets into the process. I found that if I asked:

"How do you get/set network driver parameters in Solaris?"

The answer I would inevitably get was either "I don't know" or "ndd" . Throwing the part in about gigabit makes them think through the whole process, and watching people talk through how they would do something they are unfamiliar with, even in general principles, is very valuable.  

> > 5) How does RAID-5 work? Bonus question: how does raid-4 work?
> > Extra-extra bonus question: how does raid-3 work?
>
> Most people will know that it involves stripe with parity;
> but better, I think, to ask about pros and cons of RAID-5
> versus 0+1 (or 1+0).
>

Good point. I usually use that as a follow-up question.

> > 6) What's the difference between the passwd and the shadow files?
>
> This is OK. I think I would phrase it as: What is the
> purpose of a shadow file.
>
> Answer: It's where the encrypted password is (or can be)
> stored and is readable only by root. passwd file is world
> readable. If the encrypted password is visible, it makes it
> easier to run a cracker against it to try to discover poorly
> chosen passwords.

Point taken.

> > 7) What's the difference between the dsk and rdsk devices in
> > /dev? Bonus
> > question: what's the difference between a block and a
> > character device?
>
> The high level answer involves "raw" versus "cooked" and how
> the OS accesses the device. With the complexities of OSes
> these days, I would be hesitant to try to discuss this stuff
> at the low level. Usually, if some activity requires using a
> raw device, then you are told that. As a sys admin, about
> the only time you work with raw devices directly, outside an
> application, is when you are working with disk partitions
> which, with the ascendancy of advanced filesystems these
> days, I wonder how much this is done now.

See, I'm starting to get the sense that perhaps our relative environments have brought out slightly different concepts of the role of a sysadmin. In environments where I worked, a senior Solaris sysadmin was more of an engineer - responsible for making serious architectural decisions regarding server configuration, security, storage configuration, etc. In that environment, I would absolutely expect that someone I would entrust with designing the hardware platform/configuration for our next database would be aware of the consequences of raw vs. cooked devices and take that into account.

> > 8) How do journaling filesystems work?
>
> Usually, pretty well.
>
> I think I would be more interested to know why one would use
> a JFS as opposed to UFS. I think the original reason for JFS
> was faster crash recovery (but how often does your system
> crash?). And one has to make some distinction between JFS
> and some brand of advanced filesystem. I think the terms
> tend to get smeared together sometimes.
>

Again, I see this as a situation where the asked question encompasses your question. I've never interviewed a single person that was able to give a coherent explanation as to how journaling filesystems worked that couldn't explain its advantages. Asking this question insures they know not just the advantages, but how it works.

> > 9) What's the difference between ssh and telnet? Why is one
> > preferable over the other?
>
> No comment. I guess it depends on your organization and how
> likely it is that somebody is snooping the network.
>

Sure - this is definitely where the question falls into the realm of dogma. There are security-paranoid individuals (and I am one of them) and there are non-paranoid individuals. If I'm interviewing a non-paranoid, that's fine - I just want to make sure he's giving informed consent to being non-paranoid.

> > 10) What's the difference between the e4000 and the e4500
>
> Answer: 500.
>
> (Oh, do you mean that domain thing?)
>
> I think both boxes are old models now.

They are, but there's bajillions of them out there. And its not a domain thing, its backplane speed, which is important if you're making hardware decisions. It also demonstrates that they have an awareness of sun hardware and how it works, which makes troubleshooting much easier. If they don't know sun hardware, no problem.

> > 11) What happens on an E6500 when I add boards in the bottom
> > two slots?
>
> Answer: You have to store your lunch somewhere else. Is
> fiddling with the guts of an E6500 with another one of those
> daily activities?

When you spend $50k to add two more boards to your 6500 and get zero performance improvement because of the backplane step-down, or you spend the extra 100k for a fully-loaded 6500 instead of a 6000, that's the sort of thing that's useful to know. Again, it shows an awareness of not just the OS, but the hardware as well.

> > 12) On an Sbus e-class I/O tray, what performance
> considerations do I
> > have to keep in mind when I'm installing Sbus cards?
>
> No comment.
>

Again, more hardware stuff. It's not the end of the world if they don't know sun hardware, but if they do, troubleshooting and design gets an order of magnitude easier.

> > 13) Why is NIS bad?
>
> Answer: It isn't. Depending on how your department operates,
> it might be very useful. If somebody wants to ypcat my
> encrypted password, go for it. You won't be able to crack it.
> I don't care what you do with Joe Stupid's password; that's
> Joe's problem.
>

d'oh. See security-paranoid vs. non- above.

> > 14) What's the difference between TCP and UDP?
>
> And the relevance of this to sys admin is what? Is your sys
> admin going to write a new network app and needs to decide
> what communication protocol to use? It your sys admin going
> to have a choice as to which protocol a daemon is going to use?
>

Yes, actually. What's the implications of tcp nfs vs. udp? What's the implication of tuning tcp parameters to improve Oracle listener performance? If you're just blindly setting values without knowing why you're doing it, are you actually adding value? Why doesn't the DBA just do it themselves?

> > 15) How does DNS work? Bonus question: is DNS TCP or UDP?
>
> Usually, pretty well. Is your sys admin going to set up and
> manage a domain and domain servers? It might be useful to ask
> how to tell a box it's on a domain and get it pointed at the
> domain servers. Or if a box is failing to resolve a name,
> how to check if it's using a domain server and/or of the name
> is registered with the server.
>

I've never met anyone who could tell me in-detail how DNS worked that couldn't get DNS working on any operating system.

It's a style question, as I said above. I would take 1 person who could explain how and why things work they do who had never touched Solaris (as long as they'd touched some type of *nix) over 5 people who had spent 10 years on solaris and didn't know the difference between TCP and UDP. The first person will add far more value long-term. This is, as I said previously, for senior people. For less senior, I'm fine with people who don't know the how and why, as long as they want to learn it.

It's the classic quote - give a man a fish and he's fed for a day. teach a man to fish and he's fed for the rest of his life. Give a sysadmin a book and he'll solve a few problems. Teach the sysadmin why the problem happened in the first place and he'll solve every problem Received on Wed Jul 16 2003 - 12:04:54 CDT