Linux, Unix, /etc

Danger Will Robinson! You are now entering a condescending Unix user zone!

Virtual Unix for Win32

Introduction

This article looks at the set of portable software tools provided by the Free Software Foundation"s GNU project, large parts of which have now been ported to Windows 95 and NT. They make working with Windows machines much easier, giving one a sort of "virtual Unix" environment that is very handy to work with for typical admin. tasks, and makes life easier for Unix people who find themselves having to work with Win 32 machines. Using them is often be a revelation for people without Unix experience, to whom the "software tools" approach is new and very different to the environment they are used to. In this article, we specifically look at installing the tools on Windows 95, but most of what is said is equally applicable to Windows NT.

Installation

The latest release of the software is at ftp://ftp.cygnus.com/pub/gnu-win32/latest. Installation is done using an InstallShield5—based installer. However, it is still necessary to set environment variables by hand, as explained later. The software is in two parts. firstly, there is the Cygnus Developer"s Kit (CDK), containing binutils, byacc, diff, flex, gas, gcc, gdb, ld, libg++, libstdc++, make and patch. Secondly, there are the "user tools": bash, fileutils, findutils, gawk, grep, gzip, m4, sed, shellutils, tar, textutils and time. Full source code is available for both parts. If you want the development tools and the programs necessary to run the GNU configure mechanism, you should download the whole release, which is in file called "cdk.exe". Run "cdk.exe" and follow the instructions to install the tools. The default install location is "<systemdrive>:\gnuwin32\b18". If you just want bash, fileutils, textutils, and other utilities and don"t need the development tools, you do NOT need to install the entire release. In this case, download "usertools.exe". In the configuration instructions that follow, I will concentrate on installing the whole thing i.e. everything from cdk.exe; but the configuration of usertools.exe is the same, merely skipping some extra steps that are necessary only if the development tools are installed.

Configuration

The are four GNU-win32-related environment variables to be set, without which the tools will not function correctly. These may be set in .bashrc (q.v.). If however it is required to use the tools from COMMAND.COM as well as from within bash, the variables must be added using the "System" control panel in NT or by modifying the autoexec.bat under Windows 95. Assuming the "C:\gnuwin32\b18" install location: 1) Set the GCC_EXEC_PREFIX environment variable to

C:\gnuwin32\b18\H-i386-cygwin32\lib\gcc-lib\

The trailing slash is important! 2) Add

C:\gnuwin32\b18\H-i386-cygwin32\bin;C:\gnuwin32\b18\tcl\bin

to your $PATH. This is where the executables live. 3) Set the TCL_LIBRARY environment variable to

C:\gnuwin32\b18\tcl\lib\tcl7.6

4) Set the GDBTK_LIBRARY environment variable to

C:\gnuwin32\b18\share\gdbtcl

A note on points 3) and 4). The README file that comes with the software says, "You must use forward slashes in this variable"; however, I have found the converse to be the case! As show above, use DOS backslashes. Various programs need to be able to find "/bin/sh". You should create a "/bin" if one doesn"t already exist and put a copy of "sh.exe" there. You can use the "mount" utility to select which drive letter is mounted as "/". Many programs including bash and byacc need to be able to find a "/tmp". Verify that "/tmp" exists on the drive mounted as "/", creating one if necessary. That is all the configuration that needs doing. If you should ever want to un-install the tools, you may do so via the "Add/Remove Programs" control panel. Now, lets look at what"s available.

Shell

The logical place to start is with the shell. The GNU shell (Bash, or Bourne Again SHell-—watch out, there"s plenty more bad puns where that one came from!) is a clone of the standard Unix shell, the Bourne shell, with many extensions. It acts as a plung-in replacement for COMMAND.COM. Start it by using the Run.. menu option, and entering "sh" as the program to run. You are presented with what looks like a DOS window (and that"s the icon Windows gives it). However, you have something rather more powerful. Alt-Enter will bring it to full-screen mode, which is the way I prefer to work with a command line.

Features

command line editing

The command line can be edited using the command sets of two popular (in the Unix world) editors: Emacs, or vi. The emacs set is the default; the vi mode may be used by typing the command "set -o vi". To get back to emacs mode, type "set -o emacs".

command history

A history of the commands you have entered is kept. this is in two places: a buffer in the shell itself, and a command history file. The commands to move around in the history buffer form part of the command line editing set. The command line editing commands can be used on any line in the history.

file name completion

Typing part of a file name, and then TAB, will cause the file name to be either expanded to its full name (if it is unique) or a list of matching file names to be presented (if it is not unique).

redirection

As in COMMAND.COM, a program may take its standard input from a file rather than the console by use of the "<" operator, and may put its standard output into a file rather than onto the console by use of the ">" operator. New commands may be built up from existing commands by "piping" the output of one into the input of another e.g. "sort | uniq" sorts things (lines, for instance), and discards duplicates. Unlike COMMAND.COM, support for redirection is built into the shell, and is transparent to applications, so ">", "<", and "|" will work with all command-line programs.

job control

Other programs may be started from the command line. By appending a "&" to the command, the program is started in the background, and the command prompt returns, leaving the command line free for further use will the new program is running. For example, "telnet wotan" on my local network opens a telnet window to my Linux server, leaving the command line free for further use. A background process can be killed from the shell by sending it an appropriate signal with the "kill" command. The default signal, "hangup" or "15" will cause the signalled program to terminate. But kill goes beyond this; one could start a program in the background, then send it a signal to bring it into the foreground.

Shell programming

The shell is a complete programming language. Think of DOS"s batch-file facility-—but not broken!---and then some. There isn"t the space to go into details, but the language supports a full set of structure programming constructs.

A sample start-up file

Bash reads an initialisation file on startup, called ".bashrc". This should be put in the root directory on Win95, or in the user"s home directory on NT. Those familiar with Unix should note that the file must be called ".bashrc"-—the Win32 version of bash WON"T look for .profile. Here is a listing of the .bashrc file that I use, with added comments. Note that comments in the file must be preceded by "#". For clarity"s sake, I haven"t followed this convention when adding comments for this article.

# envvars
export HOME=/

Set our home directory. This is a Unix concept that doesn"t make a great deal of sense under Windows 95; NT has a similar organisation.

export LESS="-c -C -e -g -Q"

Here, we set a special variable from which the program "less" will read command-line options when it is run.

export PATH=$PATH:.

The path is inherited from that set in autoexec.bat. Here, we take that value, and add the current directory (represented by ".") to the end. Unlike COMMAND.COM, bash does not search the current directory first, but relies solely on the value of PATH.

export PS1="donner:${PWD}$ "

The default prompt is a simple "$". this changes it to display the host name, followed by the current directory. It would be nice if we could read in whatever value is current using the command hostname, rather than hard-coding it as here, but unfortunately this is not supported under Windows 95. It can be done in NT.

# editing stuff
alias vp="vi $HOME.bashrc"
alias vv="vi -c $ $HOME_exrc"

# file management stuff
alias ls="ls -F"

Aliases are a handy feature. They work by associating a given string with a command. For example, the first line above means I have only to type"vp" to start up an editing session on .bashrc with my favourite editor. Note the use of the environment variable HOME in the alias. The "ls" alias takes an existing command, "ls", similar to DOS "dir", and sets a default argument to use. This saves us having to type out the "-F" every time

# shell options
set -o vi

There are certain shell options which I prefer to modify. Here, I set the command-line editing keys to emulate those of the vi editor, rather than those of emacs (the latter being the default).

# startup commands
cd $HOME
date

A few things to do on shell startup. Unlike Unix, we aren"t put into our home directory when our login shell starts, so we do this in the start-up script. Also, its always nice to know what day it is when you log in...

Software Tools

The power of the shell is enhanced by the host of small utility programs that can be combined with it and with each other. Here is a grab-bag of those I find most useful. The short descriptions should be enough to whet your appetite for further exploration.

Filters

The concept of the filter is a central one to the "software tools" philosophy. There is a whole set of programs whose primary function is to perform some transformation on their input, and write the result of that transformation to their output. Here are some of the useful ones available in the GNU Tools distribution:

head & last

Here is the simplest sort of filter. head takes its input and prints the first n lines to its output. Default n is 10, and can can be changed by "head -n", where n is any number. what use is that, you say? Well, there are often times when a quick glance at the beginning of a file is all you need, to find out if it is a shell script for example; again, "head -1" shows you the first line, which is useful if examining many database files ,which contain the column names in the first row. The corollary of "head" is tail, which examines the LAST n lines of a file. Tail has a special mode, "-f" for follow, which can be used to monitor a growing file-—a log file, for example-—printing new lines to standard output.

wc

A rather more complicated filter, wc is short for "word count". It writes counts the characters, words and lines in its input to its output. The options -c, -w, and -l respectively ask for only the character count, word count or line count, and can be combined.

tr

tr, short for "translate", is the classic sort of filter, where the input is changed according to a mapping. For example, to change all alphabetic characters to uppercase: "tr a-z A-Z". Or, to "rot13" some text:

    tr "[a-m][n-z][A-M][N-Z]" "[n-z][a-m][N-Z][A-M]"

sort

As the name implies, sort sorts. It has a distant relative in the DOS command of the same name, but Unix sort is far more powerful, with a wealth of options for customising field separators, default sort order, and sort keys.

sed

sed will seem a peculiar program to those used to interactive text editors. Sed is a stream editor; that is, it applies commands similar to those found in a text editor (derived, in fact, from an old Unix text editor, ed) onto a "stream" of text-—its standard input in other words. It is batch editor, and can be a very concise and powerful way of applying repetitive changes to a group of files, avoiding the tedium of making the changes by hand.

awk

Finally, we come to our most powerful filter. Awk is an interpreter for a full programming language.

Other Tools

Some other commands don"t fit into the category of filters, but are very useful.

env

env prints out your environment i.e. that collection of variables, aliases, function definitions that your shell carries around with it, and a copy of which all child processes inherit.

egrep

egrep-—Extended Get Regular Expression and Print---is a powerful tool for finding patterns in text. It searchs for text strings, but also for text patters called regular expressions. For instance, a regular expression to find a C function prototype might look like this:

    [A-Za-z_]*[^A-Za-z_]*"$1"[^A-Za-z_]*(

find

Now, this doesn"t find things IN files (unlike the DOS command of the same name), it finds files. For example,

    find /DOS -name "*.EXE"

finds all the .EXE files in the DOS directory. A very useful tool when you need to build a list of files based on some property-—last changed date, for instance-—they have in common.

Examples

OK, I"ve hit you with a lot of unfamiliar new programs, and your initial response may well be, "so what?". IT is mcuh easier to understand the power of software tools when you see them in action. Here are some examples. I"ll use features of the tools without explaining them; the full gen is in the man pages. You have a text file, and want some idea of how many pages it will take to print it.

    #!/bin/sh
    #prpages: how many pages will my printout take?
    wc $* | 
    awk "!/ total$/ { n += int(($1+55) / 56) }
	END	{ print n }"

This script works by counting the lines in all the files given it on the command line, or on it"s standard input. Then, it runs the totals through awk, which figures out how many pages that is. A quick way to produce a numbered list from a text file.

    #!/bin/sh
    awk "
    {
	gsub(/^[0-9][0-9]*[ 	][ 	]*/,");
	print ++number " " $0;
    }
    "

No pipelines or redirection here, just a simple awk script that numbers the lines in a file, or renumbers them in sequence if they were already numbered and subsequently got moved about. Here"s how to sort a bunch of lines in order of increasing length. You do this all the time, right?!

    #!/bin/sh
    awk "{printf "%d\t%s\n", length($0), $0}" |
    sort +0n -1 |
    sed "s/^[0-9][0-9]*	//"

First, get the lengths, and print them out along with the associated line ; then, sort on the length field; finally, strip off the lengths, and print out the lines in order. What date does the last Friday in this month fall on? Come on, quick! don"t know off-hand? This script tells you.

    #!/bin/sh
    /usr/bin/cal $1 $2 | 
    awk "{ lasta = a; a = $6; if (a == ") a=lasta } END { print a }"

You see how we get the output of the cal comand, which prints this month"s calendar, then check the "friday" column until there is nothing in that column. Then, the last value from the column must be the last Friday. Simple!

C Compiler

The core of the Cygnus effort is the GNU C Compiler, gcc, a portable optimising C compiler that is available on a remarkable range of platforms. There is a debugger, gdb, available as part of the CDK. Supporting he compiler is thed GNU portable assember, gas, and the linker, ld. These are called transparently by gcc when required. Applications built with these development tools have access to the Microsoft Win32 API. In addition they can use the Cygwin32 API, which provides additional Unix-like functionality including Unix sockets, process control with a working fork and select, and so on. It is possible to: Write Win32 console or GUI applications that make use of the standard Microsoft Win32 API and/or the Cygwin32 API; easily configure and build many GNU tools from source (including rebuilding the GNU-Win32 development tools themselves under x86 NT); port many other significant Unix programs to Windows NT/95 without making significant changes to the source code; and have a fairly full Unix-like environment to work in, with access to many of the common Unix utilities (from both the bash shell and command.com)

Editor

One important tool not included in this distribution is a text editor. OK, you can use Notepad or Wordpad or Word or whatever, but that"s kind of missing the point. Both of the classic Unix editors, vi and Emacs, have been ported to Win32 in a variety of forms. I haven"t the space to go into details here. check the links given in the Resources box for sources for both.

Documentation

There are two forms of documentation: standard Unix man pages, in text form; and info pages, info being the preferred documentation format of the FSF. Both forms can be found buried deep in the installed directory structure, the man pages in "man" and the info pages in "info". The documentation is obviously not a priority with the developers, being rather old and having no Win32-specific information.

Problems & Limitations

Missing POSIX functions

There"s still a lot of standard POSIX functionality that isn"t present that could be. Also, some functions aren"t implemented fully. For example, "sync" returns 0, even though it doesn"t do much.

Programs can"t deal with // pathname scheme in arguments

gcc and other tools aren"t fully compatible with the current pathname scheme: it can"t figure out an argument of -I//d/foo which means it is vital that when attempting to self-host, only normal paths with single slashes are used.

DOS special filenames

Files cannot be named com1, lpt1, or aux (to name a few); either as the root filename or as the extension part. If you do, you"ll have trouble. Unix programs don"t avoid these names, which can make things "interesting".

Conclusion

This article has introduced the reader to the many useful tools available within an environment that is a much more powerful alternative to COMMAND.COM (still, unbelievably, the command-line processor in, not only Windows 95, but Windows NT!). It should serve as a useful pointer to becoming more productive by exploiting the leverage they can add to many common system maintenacne and administration tasks.

Box: A Note on Copyright

Most of the tools developed as part of the GNU-Win32 project are GNU tools and are distributed under the GNU General Public License (GPL). Recent releases of the Cygwin32 library have also been under the GPL. In future GNU-Win32 releases (starting with beta 19), Cygwin32 itself will be licensed under a separate Cygwin32 license. Cygwin32 is used by the GNU-Win32 tools, and can be used more generally to port other Unix applications to Windows. In brief, the licensing terms for Cygwin32 permit it to be used in both free and proprietary software, except by direct competitors of Cygnus. The precise terms are spelled out in the license.

Box: Resources

The main WWW page for the GNU-Win32 project is: http://www.cygnus.com/misc/gnu-win32/ Tool-specific information can be found here: http://www.cygnus.com/library/ Links to additional documentation are accessible from the main web page. There is also the "Virtual UN*X" web page and mailing list. More info. at http://www.itribe.net/virtunix/.

Paul Dunne 1997

[back to Linux, Unix, /etc]