The joy of proper command line tools

I seem to be having lots of these problems lately that somehow always come up, but never get solved. This time it was the problem of creating a zip file including a large source code hierarchy, but excluding certain directories along the way. More specifically, there are directories called .svn, obj or bin at various places in the hierarchy, and I don’t want to include those. Although I’ve looked around many times, I was never able to find a compression tool that would allow me to specify exclusions flexibly enough to take care of this use case – go try it with your favourite packer if you think I’m wrong. Most of them have an exclusion feature, but not a single one that I found was able to handle this case.

The solution was really simple: use some command line tools. No, not stupid Windows ones, proper GNU ones. Cygwin was, of course, already on my machine, so I simply created this little bash script:

#!/bin/bash

FIND=/usr/bin/find
GREP=/usr/bin/grep

EXCLUSION='(/(.svn|obj|bin|build)/|.suo$|.user$|.exe$|.dll$|.pdb$|~$)'

while [ "$1" != "" ]; do
    $FIND "$1" -type f | $GREP -E -v $EXCLUSION
    shift
done

This script takes file or directory names on the command lines and lists all files hierarchically from those starting points, excluding those that match particular (regular expression) patterns. Simple, isn’t it? Having this as a separate script allows me to reuse it (from the Cygwin command line, of course):

./listfiles.sh Demo cheatsheet.tex cheatsheet.pdf "Workshop C#.pptx" | zip -9 complete.zip -@

And since I need to run that command regularly, I have created another little script that runs it. Done. On to something useful. How in the world do Windows users live without this sort of functionality? Are you one of them? Tell me!

Yes… I know I should have a much closer look at PowerShell to see how it can help me with this. I have actually looked at PowerShell in some depth, but from the perspective of a programmer, not a system administrator. I promise, I’ll try to find the time to see about this, and report my findings.

2 Comments on The joy of proper command line tools

  1. Lasse V. Karlsen // February 18, 2008 at 11:25 pm // Reply

    I posted a rather long little tidbit about using the subversion export command to produce a copy to a temporary directory, with only the files you’ve committed to the repository, but it got lost because your blog system loses the contents of the text field on a wrong security code (which also is a tip: try to avoid using O’s in the security code, it’s hard to tell uppercase from lowercase and I think that was my problem).As for scripting, I use python, it’s the swiss army knife of scripting.Now, let’s make a copy of this post before clicking submit.

    Like

  2. Sorry about that security code problem – I haven’t written the applet myself :-)What you mention (solving the problem from subversion) is certainly a good idea as well, but it seems a bit less general-purpose. I have the same issue when I have just created a sample project that I want to zip up, excluding the bin and obj folders.Python? Yeah, sure… I wouldn’t agree that it’s *the* Swiss army knife of scripting – there have been enough other languages around that earned that title in their time, and Python in still young 🙂 In any case, I tend to go for what I perceive to be the simplest solution. If there are a few shell commands to automate, I find it easiest to hack them into a console, try them out and finally echo them into a shell script file, make some minor modifications for automation purposes and I’m done. Perl would be my next step in many cases, i.e. when there’s just a little more processing to be done than can easily be handled with "grep"s. That’s one problem I see with more advanced scripting languages: they often move away from the basics. Shell scripting is a totally natural extension of using shell commands, which I know how to use anyway. Perl is, in many areas, just one step away from that – slightly different syntax, slightly more advanced control structures and so on – of course there’s a lot on top of that, but I’m still talking about the basic tasks. Python? I’ve used it for applications, never tried to use it as a glueing language. And there’s a similar reason why I haven’t tried using PowerShell for this kind of task yet… it sure has its own set of extremely powerful commands that can be glued together, but it’ll probably be years for the casual user to come up to the same level of efficiency with these commands that he might have achieved with Unix tools being a sysadmin at university.

    Like

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s