Erlkönig: Unix for the Intrepid

Overview

This is the syllabus for a class taught in twelve four-hour sessions spread over a three-week period. Approximately eighty-five to ninety-five percent of the syllabus is covered in a class, tailored somewhat to meet the students' interests.

Recommended Reading

Unix for the Impatient, 2nd edition (reference)
The Soul of a New Machine (historical, hardware)
Hackers, by Stephen Levy (historical)
Diamond Age, by Neal Stephenson (science fiction, computing concepts)
The Tao of Programming (humor),
Further reading...

History, Purpose, Future, Features
1. The flow of code
  1. MULTICS, UNICS, then UNIX...
  2. BCPL, B, NB, C, then diaspora...
  3. Flavors, including the AT&T and BSD variants
  4. UNIX, BSD, DARPA and the Internet
  5. MINIX to Linux
  6. User empowerment
2. Development of the OS features
  1. Shared resources
  2. Kernel as a virtual machine
  3. Multitasking/multiuser/multiprocessing
  4. Virtual memory
  5. Internet services
  6. Advantages of remote access
3. Interface progression
  1. Pre-Unix: Hollerith cards
  2. Printing teletypes
  3. Glass teletypes
  4. Intelligent terminals
  5. Bitmapped terminals and the XEROX PARC in the 1960s
  6. Suntools, the Macintosh, other early XEROX extrapolations
  7. NeWS, the NeXT, and X10 through X11R6
  8. Possibilities of the WWW, VRML, and OpenGL
  9. Research at XEROX PARC through the 1990s
  10. Textured windows in Mac OS X, c.2002
  11. Textured X (X Window System), c.2003
  12. Spatial interfaces
Initial Use of a Unix System
1. Connecting
  1. Hardwired mainframe terminals
  2. Modem connections
  3. Internet connections (more detail later under Networking)
    1. rsh, rlogin, telnet, and their security concerns (briefly)
    2. ssh
    3. making an xterm in which to start a remote session with ssh
    4. xdm
  4. The console
  5. Virtual terminals
2. Authorization
  1. login names, passwords, uids, gids, auxiliary groups
  2. selecting a login name, uniqueness
  3. login(1), su(1), sudo(1), sid(1,nonstandard), newgroup(1v,obsolete)
  4. selecting a password, obscurity
  5. passwd(1), chfn(1), chsh(1)
3. Sessions, logging in, logging out
4. Using the TERM environment variable
5. Setting up the terminal
  1. stty(1) - setup the kernel teletype driver
  2. tset(1) - automate asking what kind of terminal one is using
  3. clear(1) - clear the terminal screen
  4. shell support for the control-L for clearing the screen
6. Help
  1. the online manual - man(1)
  2. command syntax error messages
  3. command internal help
  4. desperation: using strings(1) to obtain syntax help
  5. the GNU info system
  6. the WWW and googling
  7. USENET newsgroups (now mostly overrun with spam)
  8. finding human assistance
The Directory Structure
1. Tree metaphor
2. Ancestry metaphor: parent and child directories
3. Concept of inode, directory vs. other files
4. Legal characters in filenames
5. Exploring with cd(1), pwd(1), ls(1), more(1)
6. Kind of pathnames
  1. Absolute (rooted, full)
  2. Relative
  3. Simple (no slash)
7. Pathnames as references
8. Tour - /bin /dev /tmp /etc /usr /mnt
9. Home directories
10. mkdir(1) - making new directories
11. rmdir(1) - removing empty directories
12. du(1) - check directory space usage
Files
1. Common types
  1. normal
    1. configuration files
    2. executable files
    3. READMEs
    4. dotfiles
  2. directory
  3. character special
  4. block special
  5. link
  6. pipe
  7. socket
2. ls(1) - list info about files, &c, BSD and SVr4 options
3. file(1) - try to determine the filetype by its contents
4. what(1), ident(1) - special information in SCCS and RCS files
5. Viewing files with cat(1), more(1), pg(1sysv), and less(1)
6. touch - create empty files
7. Copying and moving/renaming files
  1. cp(1) - copy files and directories
  2. multiple file copies and the target directory
  3. the BSD trailing-slash convention and its weaknesses
  4. mv(1) - move (rename) files and directories
  5. moving multiple files and the target directory
  6. copying directories with the -r option
  7. problems with cp -r, comparison to tar(1)
  8. moving directories
  9. moving things between filesystems
8. Removing files and directories
  1. hazards of removal, permanency, typing errors
  2. rm(1) - remove files
  3. rm -rf - remove whole directory hierarchies
  4. principles of shell-level wrappers around rm(1)
9. Changing Inode data
  1. chown(1) - change ownership, security & quota issues
  2. chgrp(1) - change group
  3. chmod(1) - chmod mode and permission, octal and symbolic
  4. chattr(1) - extended file attributes under linux-like systems
10. ln(1)
  1. hardlinks vs symbolic links
  2. creating new hardlinks
  3. creating new symbolic links with the -s option
  4. quirks of symbolic link permissions
11. Creating specials and pipes with mknod(1)
Shells
1. Interface between user and kernel
2. Shells as programming languages
3. Writing Unix shells
4. Existing Unix shells
  1. Bourne at AT&T
    1. sh - programming
    2. jsh - job control
    3. ksh - command edit
  2. Csh at Berkeley
    1. csh - history
    2. tcsh - command edit
  3. Plan9 at AT&T
    1. rc - lisp-like
    2. es - extensible
  4. Hybrids
    1. bash - GNU's ksh/csh superset
    2. zsh - ksh/csh superset
  5. Other metaphors
    1. ash - the adventure shell
    2. Graphic interfaces
5. Shell configuration
  1. configuration files, backwards compatibility
  2. env(1), set(1), vars(1rc), echo(1), shopt(1,bash)
  3. local vs exported variables
  4. setting local and environmental variables
  5. Specific environment variables
    1. PATH - the command search path
      1. finding command directories to add
      2. dangers of having "." too early in the path
    2. MANPATH - the manual search path
      1. finding manual directories to add
    3. PAGER - customizing the viewing of long files
    4. TERM - setting with tset(1) or directly
    5. CDPATH - search path for cd, in shells with support it
    6. LD_LIBRARY_PATH and kindred
  6. setting the prompt string, $PS1 or $prompt
  7. reloading the environment
  8. functions, aliases
  9. problems caused by masking system programs
6. Command Syntax
  1. command names and arguments
  2. multiple commands on a line using semicolons
  3. input/output redirection
  4. background jobs with ampersand
  5. pipelines
  6. using tee(1) in pipelines
  7. delayed execution with sleep(1)
  8. exit status and how to see it
  9. using filters and sponges
  10. substitution and rewrite
    1. $ - variable
    2. ``, $() - output
    3. ~ - tilde, home directory
    4. ! - history
    5. \, '', "" - quoting to prevent certain substitutions
    6. filename wildcards
      1. * - any number of any character
      2. ? - one of any character
      3. [] - one of a set of characters
        
        [abcd] - sets
        [a-d] -ranges
        [A-Za-z_-] - combined sets and ranges
        [^0-9] - invert sets
    7. special treatment of . in dotfiles
    8. {} - filename expansion
    9. filename expansion to produce wildcards
    10. process substitution (in bash and zsh) with <(cmd)
7. Editing command lines
  1. the kernels teletype-driver interface
  2. history substitution
  3. commandline edit in Emacs mode
8. Job control in the Csh family
  1. jobs - list jobs, and process ids with -l
  2. kill - using named signals, and % with job ids
  3. bg - make a job resume in the background
  4. fg - make a job resume in the foreground
  5. using control-z to suspend foreground jobs
9. Programming constructs in the Bourne shell family
  1. do, done
  2. for
  3. true, false
  4. if, then, elif, else, fi
  5. case, esac
  6. the null command using a colon
  7. shell scripting and the user's own bin directory
  8. scripts vs functions/aliases
  9. writing a program which counts from 1 to 100
  10. getting input with read
10. Programming with Unix system commands and idioms
  1. slowing loops using sleep(1)
  2. test(1) and its other name, [(1)
  3. expr(1)
  4. comparison between expr(1) and bc(1)
  5. free input handling using cat(1)
  6. verifying input with grep(1)
11. Creating shell scripts
  1. adding $HOME/bin to PATH in the ~/.profile or equivalent
  2. placing script files in the ~/bin directory
  3. the interpretor line
  4. shell code
  5. ensuring the final newline with comments
  6. setting the execute bit with chmod(1) and -x
Processes
1. Concept of a process
2. Fork-and-Exec process creation
3. How a shell...
  1. runs a given command and waits for it
  2. runs a given command in the background with &
  3. does input/output redirection
  4. sets up pipelines
4. Inheritance from the parent
  1. user, group, and auxiliary group ids
  2. input/output descriptors, example made of getty(1)
  3. environment
5. Process control
  1. priority - nice(1), renice(1)
  2. signaling - kill(1), and its -l option
  3. resources - ulimit(1sh)
System Status and Account Information
1. Commands you can only run on yourself
  1. whoami(1), logname(1) - find your username
  2. quota(1) - are you over your file quota?
  3. id(1) - output your current access rights
  4. tty(1) - find your terminal's device file
2. Getting info about other accounts
  1. w(1), who(1), finger(1) - find who's logged in
  2. last(w) - who's been logged in historically
  3. fingering other users
  4. fingering another machine's users
  5. groups(1) - find a user's group access rights
3. Time
  1. the TZ variable
  2. date(1) - output the current time in sundry formats
  3. uptime(1) - how long the machine's been up, and the load
  4. ntpq(8) - co�rdination with Universal Time
4. System activity
  1. ps(1) - what processes are running, BSD and SVr4 variants
  2. lastcomm(1) - list programs people have been running
  3. netstat(1) - current network connections
  4. df(1) - filesystem list and current disk usage
  5. top(1) - topmost active process list, time, and memory stats
Editors
1. Buffered editors vs. stream and direct editors
2. Line editors vs. character editors
3. Ed
  1. history
  2. basic editing capabilities, using: . $ p a i w q d
  3. cautions, getting more descriptive errors with H
  4. in-place editing through careful scripting
    1. the q, w, q sequence
    2. comparison with: perl -pi -e '....'
4. Vi
  1. history
  2. basic editing capabilities, HJKL, etc.
  3. modality
  4. cautions and recommended practice
  5. vilearn(1)
  6. comparison with newer versions
5. Emacs
  1. history
  2. basic editing capabilities, using Control and Meta, etc.
  3. modality
  4. cautions and recommended practice
  5. the emacs tutorial
Filters and Sponges
1. Filters vs sponges and immediacy of output
2. Basic filters
  1. tr(1) - translate or delete characters
  2. grep(1) - selecting lines
  3. uniq(1) - removing and counting repeated lines
  4. nl(1) - number lines in a file
  5. bc(1) - esoteric uses of the basic calculator
  6. cut(1) - keeping only certain character-columns
  7. rev(1bsd) - reverse lines left-to-right
  8. head(1), tail(1) - retain only beginning or end of output
  9. using compression programs in filter mode
3. Sponges
  1. sort(1) - sorting lines, possibly numerically
4. Programmable filters
  1. sed(1) - stream editor, string replacement
  2. awk(1) - interpreted language, accessing fields
  3. perl(1) - interpreted language, full-featured
  4. m4(1) - a hairy macro language
Communications
1. Realtime interuser communications
  1. writing text directly to a terminal device file
  2. using mesg(1) and write(1)
  3. talk(1) and ytalk(1tal)
  4. irc(1) - Internet Relay Chat
2. Electronic mail
  1. Addressing
    1. within a site
    2. uubang addressing
    3. Internet addressing
    4. determining your own email address
    5. finding someone else's email address
      1. name uniqueness
      2. using address search sites like www.four11.com
    6. aliases, .forward file, and the vacation(1) program
    7. contacting a site's Postmaster
  2. Transmission and access
    1. SMTP - simple mail transfer protocol
    2. message forgery under SMTP
    3. POP - pop office protocol
    4. IMAP - Internet message access protocol
    5. security problems due to cleartext transmission
    6. better security through mechanisms such APOP and IMAPS
    7. web-based access to server-side IMAP manipulation
  3. Composition and reading
    1. traditional - mail(1), biff(1)
    2. the Rand MH mail system
    3. graphic user agents
    4. POP agents and how to configure them
    5. filing mail, including the filing of outgoing mail
    6. signature files
    7. computer-to-human email communications and mail(1)
    8. MIME - multimedia Internet mail extension
  4. Mailing lists and majordomo(8)
  5. Security
    1. misaddressing and misdirection
    2. spam and mailbombs
    3. spam reduction
      1. email address concealment
      2. sendmail /etc/mail/access database configuration
      3. spam sender verification using traceroute
      4. the Realtime Blackhole List
      5. spamassassin
    4. problems with reverse spamming
    5. using PGP - pretty good privacy
3. USENET News
  1. genesis of the USENET news system
  2. NNTP - network news transmission protocol
  3. finding an NNTP newsserver
  4. reading news via an NNTP client program
  5. group overview, special status of alt.*
  6. normal and moderated groups
  7. signal-to-noise ratios
  8. why the WWW has largely supplanted USENET
Utilities
1. find(1), xargs(1)
  1. selecting a starting directory
  2. the -print directive
  3. locating anything matching a name or (quoted) wildcard
  4. locating only certain types of files
  5. locating files owned by particular a user or group
  6. locating files of a certain age
  7. using find(1) in combination with xargs(1) and other commands
2. wc(1) - count words, lines, and characters in files
3. diff(1), diff3(1) - find differences between text files
4. comm(1) - find common and unique lines in a pair of sorted files
5. cmp(1) - comparing binary files
6. cal(1) - output a calendar
7. od(1) - octal dump
8. join(1) - join files into columns
9. yow(1), fortune(1) - text generators
10. vacation(1)
11. units(1) - conversions
12. script(1bsd) - make typescript of terminal session
13. spell(1), ispell(1gnu) - spelling checkers
14. look(1) - lookup words or lines in a file by prefix
15. strings(1) - dig readable strings out of a binary file
16. leave(1) - remind yourself to leave the computer
17. whois(1) - find Internet-significant persons
18. sum(1) - summarize a file into a checksum
Document Formatting and Printing
1. Comparison of markup, languages, and WYSIWYG editors
2. Basic formatting
  1. fold(1), fmt(1) - wrap text to a certain line maximum
  2. pr(1) - paginate
3. Languages
  1. The ROFF document formatting package
    1. nroff(1), troff(1), groff(1) - typeset
    2. deroff(1) - remove Roff typesetting codes
    3. col(1) - remove char-sequence that confuse terminals
  2. TeX and LaTeX
  3. PostScript
4. Markup
  1. HTML - hypertext markup language
  2. SGML - standard generalized markup language
  3. XML - extensible markup language
5. WYSIWYG
  1. FrameMaker - document typesetting in WYSIWYG form
  2. HTML editors - what you see is -all- you get
6. Printing
  1. Translating documents to understood printer format
  2. BSD: lpd, lpr, lpc, lpq, lprm, /etc/printcap
  3. SYSV: lp, enable, cancel
Spooling
1. Overview of spooling and related directories
2. Services using spooling
  1. Printing
  2. News
  3. Email
  4. Paging (on applicable systems)
Software Development
1. Overview of the programming process
  1. Machine language
  2. Assembly language and assemblers - as(1)
  3. Human-readable languages and compilers - cc(1), cpp(1)
  4. Linking - ld(1)
  5. Superlanguages - yacc(1), lex(1) (or bison(1gnu) and flex(1gnu))
  6. Debuggers and core dumps - dbx(1), gdb(1gnu)
2. Source management and editing
  1. special modes in Emacs for editing source code
  2. indent(1), vgrind(1) - source formatters
  3. make(1)
  4. SCCS(1), RCS(1), and CVS(1)
  5. patch(1) and context diffs made with diff(1) and -c3
  6. cxref(1) and cflow(1)
3. Investigation of compiled code (machine code)
  1. ldd(1) or elfdump(1sgi)
  2. strings(1)
  3. nm(1)
4. Investigation of running code
  1. ps(1) to get the process id
  2. trace(1), SGI's strace(1), or Linux' ptrace(1)
  3. debugging running code with dbx(1), gdb(1gnu) or xxgdb(1gnu/x)
  4. the /proc filesystem
5. Special issues related to system kernel configuration
  1. configuring a kernel to machine system hardware
  2. configuring a kernel for certain software capabilities
  3. configuring a kernel to allow for usage or optimizations
  4. rebuilding a system
  5. holding a backup kernel
  6. historical move toward more flexible and dynamic kernels
  7. kernel debuggers - adb(8)
Software Importation and Compilation
1. Where to get software on the Internet
2. The benefits of source code
3. Acquiring source code
  1. finding software on the WWW
  2. finding software using archie(1)
  3. using ftp(1) to transfer files to your computer
  4. unpacking the distribution
  5. shar(1), unshar(1) - shell archives
4. Compiling source code
  1. finding the README and INSTALL files
  2. GNU project software and ./configure
  3. X software and xmkmf(1)
  4. using make(1) directly
  5. editing Makefiles
  6. editing sourcecode
  7. final installation
5. Making distributions with tar(1) and gzip(1)
Compression and Archival
1. pack(1v), unpack(1v)
2. compress(1), uncompress(1)
3. gzip(1gnu), gunzip(1gnu)
4. zcat(1) - generalized program for decompressing to stdout
5. ar(1) - archiver
6. tar(1) - tape archiver
7. problems with older tar(1) implementation and SGI's bru(1)
8. Storage formats
  1. tape - paper, reel, QIC, 4mm, 8mm, DLT
  2. other - CDROM burning, removable disk media
9. manipulating tapes with mt(1)
10. purging spurious files before archival with find(1)
Encryption
1. conventional
  1. rot13
  2. substitution
  3. DES
  4. one-time pads
  5. modern approaches
2. public key encryption with PGP - pretty good privacy
  1. private key
  2. public key
  3. key ring
  4. encryption
  5. decryption
  6. signatures
  7. integration with electronic mail
3. risks of public-key escrow systems
Networking
1. Unix networking
2. Pre-Internet
  1. issues of intermittent connectivity and resource contention
  2. modem-type hardware
  3. nodenames
  4. cu(1), tip(1), and the unix-to-unix command set
  5. sz(1), rz(1) - Zmodem communication over old-style phone links
3. Internet
  1. issues of continuous connectivity
  2. networking hardware
    1. ifconfig(8) - display/find network local interfaces
    2. likely connected hardware and uplinks
    3. modems vs. ISDN vs. T1/ADSL, etc.
  3. IP addresses
  4. ping(1) and traceroute(1)
  5. hostnames
  6. domainnames
  7. DNS - domain name service
    1. the old hostlist on sri-nic.arpa, later nic.ddn.mil
    2. problems encountered with increased Internet size
    3. DNS servers and specifying them in configuration files
    4. nslookup(1) for IP addresses
    5. other nslookup commands
  8. routing
    1. how routing works
    2. default routes and specifying them in config files
  9. ftp(1) and the ~/.netrc
  10. telnet(1)
  11. rlogin(1), rsh(1), rcp(1), and the ~/.rhosts file
  12. ruptime(1), rup(1), rusers(1), rwall(1), rwho(1)
  13. issues of packet visibility and security concerns
  14. ssh(1), scp(1), and the ~/.ssh/ directory
Administration
1. Principles of systems administration
2. The root user
3. Authentication - passwd(1), passwd(5), group(1), login(1)
4. Halting - shutdown(8), halt(8), reboot(8), init(8)
5. Understanding SCSI - small computer systems interface
  1. the SCSI bus and device count limits
  2. SCSI-1 through SCSI-3
  3. fast, wide
  4. uses for fast wide SCSI
  5. single-ended vs differential
  6. advantages of multiple SCSI busses
6. Disks
  1. format(8), SGI's fx(8)
  2. disk labels and partitions
    1. stupid BIOS limitations and virtual partitions
    2. subdevices and limits on SCSI drives
    3. reasons for using partitions
      1. risk reduction vs block failure
      2. damage containment
      3. partitions as backup units
      4. speed issues when used for swap
      5. politics
  3. unix filesystem formats
  4. journaled filesystems such as XFS (on SGI), an EXT3 (on Linux)
  5. newfs(8), mkfs(8)
  6. mount(8), umount(8)
  7. filesystem exportation
7. Terminals - ttys(5), ttytab(5), termcap(5), terminfo(5)
8. wall(1)
9. Message of the Day - /etc/motd
10. Accounting - sa(8)
11. Unix runtime commands - /etc/rc*
12. The problem of a lack of dependency checks in the /etc/rc subsystem
Power Users
1. Group-sharing of files and directories using chgrp(1) and chmod(1)
2. Scheduling - crontab(1), at(1)
3. mkfifo(1), mknod(1), and the use of FIFO pipes
4. find(1) and the -o and -prune options
5. Heterogeneous networks and the command search path
6. Heterogeneous networks and effects on configuration files
7. Setting up configuration files for multiple shell types
8. screen(1) - using detachable sessions
9. rdist(1)
10. Controlling processes remotely using kill(1) with SIGSTOP
11. Interfacing with wireless alphanumeric paging devices

Erlkönig: Unix for the Intrepid

Overview

Recommended Reading

History, Purpose, Future, Features

Initial Use of a Unix System

The Directory Structure

Files

Shells

Processes

System Status and Account Information

Editors

Filters and Sponges

Communications

Utilities

Document Formatting and Printing

Spooling

Software Development

Software Importation and Compilation

Compression and Archival

Encryption

Networking

Administration

Power Users