Unix script help needed

[selenay]

I have hit a brick wall and any help that can be offered would be gratefully appreciated. My Google-fu isn't up to the task, we have no reference books on Unix scripting here ("here's a card with basic Unix commands - have fun!"), my local library has no books on Unix scripting and I haven't got five days for Canada Post to lose deliver an Amazon order of books for me.

I'm attempting to loop through some files in a directory, count the number of files that contain a particular word sequence and put that count into a file that I can then email. All the filenames that I am searching begin with 'O'. So far, if I cd to the directory and put this into the command line it works:

print cat O* | grep -c 'XXXX_PROD' >> test.log

My log file correct shows a count of 2.

When I do the same thing in my script and run it, I get big fat zeros in my report :-(

cd $LOGDIR1
print $PWD >> $REP
print "Count of XXXX_PROD jobs run (test only):" >> $REP
print cat O* | grep -c 'XXXX_PROD' >> $REP

According to my log, I've cd'd to the correct directory. I've tried using $LOGDIR1/O* with no luck.

Can anyone point me to some resources that might help? If it helps, I would appear to be using the Korn shell in this installation of Unix. The mere fact that I've had to write "appear to be using" should tell you how much I actually know about Unix :-(

Comments

[tejas.livejournal.com]

Did you check the man pages on your system?

[selenay]

This is where I betray my complete lack of knowledge. What are man pages?

[tejas.livejournal.com]

Not a problem. While I'm not a coder, I've worked on UNIX systems for years. Man pages are your friend. They're the online manual for UNIX. At the prompt type [man help] (or way, maybe it's help man, been a while). That should give you the necessary intro into how to use the manual. You might be able to find what you need there... or at least better search criteria for google.

[paranoidangel]

Man's great, although at uni someone had written some code that meant that typing woman did the same thing :)

(I know how to use Unix since the Unix machines at uni were better than the Windows ones, I just don't know how to program in it)

[selenay]

I can use basic Unix stuff so that I can browse around the system, read files, search for stuff in files etc. The most I've previously had to do is amend scripts to look for things in different directories. Actually building a new script is a whole new thing for me.

Apparently my boss thinks I know more about programming than I actually do. Perhaps she's been fooled by my kick-ass SQL skills into thinking that I can handle anything?

[nic]

I only know pieces of Unix, but:

- What scripting language are you using? i.e. what kind of file is it?

- Is there some kind of 'end of line' character that needs to go in there?

Just guessing, sorry, but the above might prompt a thought?

[selenay]

I have absolutely no idea what kind of file it is *sigh* No file extension, no clues. I've basically just copied a file from another job and attempted to plunk in the stuff I need.

I am starting to suspect that what I need to do is beyond the levels of what I know. And what I can actually usefully find out from my colleagues.

[gmul.livejournal.com]

You may have it sorted with the pointer towards 'man' from tejas. I'm not sure if you absolutely need the cat - you might be able to get enough with just something like
grep -binary-files=text -c -r 'XXXX_PROD' *
(not that I'm sure that's an entirely valid command...)

Or

You may not have the cat and grep around the right way. You're feeding the output of cat in to grep, rather than the other way around which I think is what you'd want, no?

As you can tell this isn't really my thing either...

[gmul.livejournal.com]

No, forget the Or part of that at least for now...

Just grep 'XXXX_PROD' O* might do the trick - give you a list of files that have a match to the pattern in. Then you just have to count the number of lines somehow.

[selenay]

The cat/grep combo works fine when I type it into the command line, it's when I put exactly the same thing into a script that it all falls over. And I have no idea why :-( The cat was the only way that I could find (using Google-fu) to get the count of the number of lines rather than a huge list of file names with a count for each.

This really isn't my thing. And my boss cheerfully expects me to have the entire script tested and ready to go by end of play today. Um, I think not, somehow.

[gmul.livejournal.com]

I'm still not sure cat is counting the right thing but that's probably my understanding.

Let me see who's in along the way, I'm sure my neighbour Dave will know and he was in earlier... Back soonish!

[gmul.livejournal.com]

The committee here also came up with the wc mentioned below, we just spent too long talking about bikes for me to get back and post it. :-)

[selenay]

Heh :-) Thanks for trying - at least you got a fun afternoon out of it!

[cobrabay.livejournal.com]

It is possible that your script is running under a different shell to the one you are using when you type the commands in yourself, resulting in commands having different effects.
How are you running your script, and what is the first line of the script? If you type

echo $SHELL

that will show what shell you are using on the command line. If you are using the Korn shell, that should give

/bin/ksh

The first line of your script should contain a line something like the following,

#!/bin/ksh

If not, then the shell used will depend on how you run the script. If you are just typing in the full name of the script, then it will default to your login shell (presumably korn shell).
Your command could be changed (there are always many ways to do the same thing in a Unix shell), the syntax for grep is,

grep [pattern to find] [file(s) to search]

So try this,

grep 'XXXX_PROD' O*|wc -l >> logfile

This will find all instances of XXXX_PROD in all files starting O*, pipe the results into wc and count the lines. The output number is your total.

[cobrabay.livejournal.com]

Update: I forgot to add that this method only works of you only have one instance of XXXX_PROD per line.

I'm not sure why you are using

print cat O*

, as just

cat O*

should work.

[selenay]

I had just been starting to wonder whether I was dealing with different interpreters but had no idea how to check. When I query the command line, I get /usr/bin/sh returned.

The first line of the script that I've hacked is #!/bin/ksh.

Sounds like this means I'm using Korn in the script and something different in the command line, which would explain the differences. At least I'm not crazy!

I've just tried your grep suggestion (including removing the echo/print) and it works. My log file actually finally contains the count that I was expecting. I should only have one instance of 'XXXX_PROD' per line. If I have more than that then something funky has happened (I'm trying to automate daily checking of job logs) that needs investigation and thus it's important to see that weird count anyway. If that makes sense.

The wc command rings a bell from when I was attempting to work out the basics of Unix last year, but I'd totally forgotten about it until now.

The print cat O* is an example of just how clueless I am about this stuff and how much I was relying on hacking bits of existing scripts with little knowledge of what I was doing! It's starting to slowly make a little more sense.

I really need to get myself a Unix reference book if I'm going to get given these 'challenges' regularly *sigh* I'll toddle off to Amazon tonight, but at least there's a chance that I'll have a proto-type script ready for Da Boss today. Thank you so much.

[cobrabay.livejournal.com]

The O'Reilly books are pretty good for this, "Unix in a Nutshell" is a classic. Their "Learning the bash Shell: Unix Shell Programming" is pretty good too if you have to write many shell scripts, n.b. bash is more feature rich and newer version of sh, available on most Unixes, and all Linuxes, (and I much prefer it to Korn shell), but there is enough overlap to make this book generally useful for shell programming.

[sugoll.livejournal.com]

Since you said you wanted count of files, rather than count of lines, then:

grep -l PROD_XXXX O* | wc -l

grep -l will list the file name that match; wc -l will count them.

Note: you said "word sequence". It gets more awkward if you want to match something with whitespace, or punctuation that might be interpreted by your shell.

[selenay]

Ooh, even better! I have got to get me a decent reference book or two. Thank you :-)

It will only ever be a word or two joined by underscores - I'm basically looking for schedule names in a couple of hundred log files each day so that I can find out how many jobs ran under each schedule and also checking for a standard error flag. We want to email a report each morning rather than having someone spending half an hour manually checking all the logs. Someone will still have to read the report and investigate discrepancies, but at least we'll cut out that thirty minutes a day of hunting for the problems!

[sugoll.livejournal.com]

Good news that it's not a pattern with whitespace.

I trust all the files are in the same directory? If they're scattered through subdirectories, then "man find" is your friend.

Incidentally, on the subject of reference books: the basics of this stuff haven't changed much since 1970. There are lots of bells and whistles added, but the core points of shell programming are still valid. Which means you might be able to pick up a basic introductory reference that's a couple of decades old for next to nothing, in second-hand bookshops around an academic establishment. Once you have the basics, then you can dig around a lot on your current system using "man", which will save on the cost of a hefty new tome.

[selenay]

Two different directories, but I'm searching for different job names in each directory so there's no searching through different sub directories.

Hmm. I might have to investigate some of the second hand shops around Dal - having the basics at my finger tips would be good and then I'd at least have a better idea of what to look for when I'm trying to find other, less-basic stuff.

I know SQL so well and felt so proud of my skills. Trying to play with shell scripting is reminding me of how much there is out there to learn! Still, by the time I'm done with this stuff request I'll have learned some new things for my resume :-)

And then I get to do this all over again with Clipper, apparently. At least there is a reference book in the department on Clipper. Although nobody in the department knows anything at all, even less than they know about shell scripts!