El Cazador-Recolector™

September 9th, 2005

Leo un artículo en Versión Cero que contiene el siguiente párrafo:

Significativo es el caso de la historia contada por un antropólogo en su libro sobre los pigmeos. Recuerdo vivamente esta historia. Fueron todos a cazar, incluido el antropólogo. Y lograron cazar un gran buey salvaje. Mientras iban de camino a la aldea con la pieza cobrada, iban despreciando al cazador y el resultado de la jornada:

- “era un buey viejo, seguro que su carne está dura”

- “con esta pieza no tendremos para alimentar a todos”

El antropólogo, perplejo con lo que oía, pues la pieza bien podía significar el alimento de toda una semana, preguntó a uno de ellos porqué despreciaban ese ejemplar y más concretamente al cazador cuya intervención permitió cobrarlo. Su contestación fue: “si alabamos al hombre por cazar, crecerá su orgullo, pues los hombres son orgullosos por naturaleza, y pensará que es mejor que los demás, creando conflicto y peleas entre nosotros. El orgullo lleva a la envidia, la envidia lleva al odio, y el odio lleva al crimen. Nadie es mejor que nadie, todos somos iguales. Así que hay que despreciarle para que su orgullo no crezca.”

Realmente no sé si esta tribu de pigmeos realmente dijo eso o no, pero esa frase encierra una gran sabiduría y comprensión de la auténtica dimensión humana.

Incremental backups with rsync

September 9th, 2005

I have been thinking for a while to implement incremental, cyclical backups on my home network. The problem with cyclical backups to tape is that they are slow. The problem with cyclical backups to disk is that they consume a great deal of space. I finally opted for cyclical backups to disk since my DDS-3 SCSI tape is slow and can’t hold the many gigabytes I have in data, even with hardware/software compression.

I want to periodically branch my main backup tree so that I can keep several backups, ordered from the newest (backup.0) to the oldest (backup.n), where “n” could be the number of days or weeks, depending on the frequency of the backups.

The filesystem should look like this:

\-- backup.0
 |
 |- backup.1
 |
 |- backup.2
 |
 .
 .
 .
 \- backup.n

A simple way to reduce disk space usage is by using a UNIX-like feature called hard-links. The idea behind this is that if a file does not see its contents changed between backups, we could save space by having all the identical copies hard-linked together.

Using rsync and cp we can implement this very easily, thanks to the way that rsync works. By default, when not using the –inplace command-line switch, if rsync detects that a destination file is different from its source file, instead of performing direct modifications onto the destination file by opening it, writing to it, then closing it, rsync will create a new file. This has several advantages:

  1. Users can keep on working with files, even when rsync is synching them underneath. Since rsync always creates a new file instead of performing modifications to the current file, users won’t suffer from the strangeness that involves multiple updates to the same file by multiple users/processes.
  2. Since rsync creates a new file, when the original destination file is hard-linked across several backup branches, the synching process won’t indirectly sync up those backup branches too. Instead, they will be kept intact, and a new destination file, mirroring its source file, will be created.

    We don’t want that an update to a file in the backup.0 branch means updating any file hard-linked to it, since that would destroy the incremental semantics.

Thus, we can implement a really simple cyclical backup scheme using rsync and hard-links..

  1. Things to run on the server.

    We run this periodically:

    # rm -fr backup.${n}
    # for i in `seq ${n} -1 2`; do mv backup.$[${i}-1] backup.${i}; done
    # cp -al backup.0 backup.1

    This will rotate all the backups, discarding the last one. Then, the cp command will replicate the main branch (backup.0) into (backup.1) by using hard-links.

    NOTE for FreeBSD users: the cp command that comes with the FreeBSD base system does not support neither the -a nor the -l command-line switches. -a means -dpR (recursively copy and preserve attributes), while -l means not to copy, but to create hard-links instead.

    Fortunately, the FreeBSD ports collection includes a port of the GNU coreutils package, which sports the full GNU cp program, supporting the -a and -l switches:

    # cd /usr/ports/sysutils/coreutils
    # make all install

    To avoid the name clashing betweeh the cp command from the FreeBSD system and the GNU one, the GNU cp command is renamed to gcp. So, in the script listed bedore, we should rename the invocation to cp to gcp.

  2. Things to run on the client.

    To perform the incremental backup against the server, we can run the following command:

    # rsync -a -E Users/ rsync://:/data/backup.0/

    It’s very important to keep the timestamps synchronized on both the client and the server so rsync can use them to decide which files have been changed and which files not. This is done with the -t command-line switch. Note that the -a (archive) command-line switch to rsync is like specifying -rlptgoD, and thus we don’t have to specify -t.

    The -E command-line switch is useful for Mac OS X-based machines and will allow synching files stored in a HFS+ volume that uses resource forks by using the AppleDouble format.

The rsync software that comes with Mac OS X 10.4, and newer releases, supports extended attributes (HFS+ resource forks). This means it can sync files from a local HFS+ filesystem to a remote volume which does not suport HFS+ resource forks by using AppleDouble encoding.

The AppleDouble encoding splits a native HFS+ file in two parts:

  1. The data fork, which is the one that holds the real contents of the file, like the contents of a document, the pixels from a bitmap, and so on. It receives the same name as the original HFS+ file.
  2. The resource fork, which is built of data held on the resource forks and Finder data, like the Spotlight comments and so on. It receives a filename which consists on prepending a dot and a slash characters to the original HFS+ filename.

Thus, for a HFS+ file named MyDocument.webloc, when stored in the AppleDouble format, it is splitted in two files: MyDocument.webloc and ._MyDocument.webloc.

By default, Mac OS X rsync implementation does not enable extended attributes support. This must be explicitly enabled by supplying the -E command-line switch to the rsync command. The problem is, however, that few rsync implementations (I don’t know of any besides Apple’s Mac OS X 10.4 one) support neither this kind of functionality nor the command-line switch that activates it.

The solution was pretty easy, by the way. I downloaded Darwin rsync source code for the rsync-20 from the Darwin Projects Directory and extracted the file patches/EA.diff from within it. This patch file includes the extended attributes (HFS+ resource forks) functionality I was seeking. This patch, at the moment of this writing, is agaist rsync-2.6.3.

Thus, I only had to grab the sources for rsync-2.6.3, which are also included inside the rsync-20.tar.gz file I downloaded before, extracted them, patched, configured, made and installed:

# tar zxvf rsync-20.tar.gz # cd rsync-20 # tar zxvf rsync-2.6.3.tar.gz # cd rsync-2.6.3 # patch < ../patches/EA.diff # CFLAGS="-O -pipe" \ ./configure --prefix=/usr/local \ --disable-debug --enable-ea-support \ --with-rsyncd-conf=/usr/local/etc/rsyncd.conf # make # make install

I ran the previous commands on FreeBSD 7.0-CURRENT, thus the /usr/local prefix. Also, note the –enable-ea-support command-line switch supplied to configure. It is required in order to build the extended attributes support in. Leaving it out will produce a normal, EA-disabled rsync.

# rsync --help | grep -- -E
 -E, --extended-attributes   copy extended attributes

That’s all, folks. :-)

Static routes are stored in a NVRAM variable called static_route. This NVRAM variable is a blank-delimited list of static route entries. Each entry has the following format:

SUBNET_ADDRESS:SUBNET_MASK:GATEWAY:METRIC:INTERFACE
  • SUBNET_ADDRESS is the IP subnet address, calculated by applying the netmask to the gateway address, for example.
  • SUBNET_MASK is the subnet mask used to disguise which part, of a given IP address, corresponds to the subnet and which part correspond to the host.
  • GATEWAY is the IP address of the gateway used to reach the subnet.
  • METRIC is the route metric. Routes to the same destination with lower metrics are preferred over those with higher metrics. So, this is usually set to 1 for single-path routes.
  • INTERFACE is the interface name used to reach the GATEWAY. Usually it is not needed, as this can be disguised from the current routing table, but the WRT54G implementation of static routes requires this. It is usually vlan0 for LAN-reachable gateways.

During the next boot, the /etc/init.d/S40network initscript will read the static_route NVRAM variable and will inject static entries into the routing table derived from it.

Linksys WRT54G (cont.)

September 7th, 2005

I have been playing around with the WRT54G a little bit more.

Instead of acting as a Wireless AP and forwarding traffic to the Internet via the WAN port, I have configured it to forward packets coming from the LAN (via any of the 4-LAN ports) to the Internet via the Wireless interface. The WRT54G will associate with an existing Wireless AP and will use it as the gateway to forward traffic to the Internet.

These are the steps I followed to configure WRT54G in such a way:

  1. Break the bridge between wired and wireless interfaces:

    In its default configuration, the WRT54G router is configured to bridge together the wired (vlan0) and wireless (eth1) interfaces. Since the WRT54G is being configured to route between the wired interface (LAN) and the wireless interface (Internet), this bridge must be disabled:

    # nvram set lan_ifname=vlan0

    Setting lan_ifname to br0 will bridge together the interfaces specified by lan_ifnames under a bridge interface named br0. Setting lan_ifname to vlan0 means no bridge will get built up, and that LAN traffic will flow through and from the vlan0 interface.

    NOTE: vlan0 represents the 4-LAN ports of the WRT54G, vlan1 the WAN port, and eth1 the Wireless interface, by the way.

    # nvram set lan_ifnames=vlan0

    This variable defines the interfaces that will get bridged together into br0. Since only one interface is listed, and lan_ifname is not set to br0, no bridge will be built.

  2. Configure WRT54G as a Wireless client:

    The Wireless interface will behave like a WAN port. Thus, I have used the wan_* NVRAM set of variables to configure it. I run the following commands from inside a router shell:

    # nvram set wl0_mode=sta

    This configures the WRT54G to act as a Wireless client, instead of a Wireless AP.

    # nvram set wl0_ssid=

    Configures the ESSID of the Wireless LAN the WRT54G will try to associate with.

    # nvram set wl0_wep=enabled

    WEP authentication is required in order to associate to the Wireless AP.

    # nvram set wl0_key=1

    Use the first WEP key, of the four available WEP key slots.

    # nvram set wl0_key1=

    This fills in the first WEP key slot with the correct key.

    # nvram set wan_proto=static

    Use static IP configuration (no DHCP).

    # nvram set wan_ipaddr=

    Configures the IP address for the wireless interface (acting as the WAN port).

    # nvram set wan_netmask=

    Sets the network mask for the wireless interface (acting as the WAN port).

    # nvram set wan_ifname=eth1

    We tell the initscripts that the Wireless interface will act as the WAN port (outside, firewalled, Internet connection).

    # nvram set wan_gateway=

    Defines the gateway that will be used as the default route to reach the Internet.

    # nvram set wan_hostname=

    Sets the router’s FQDN, like linksys.local or openwrt.example.com.

    # nvram set wan_dns=

    Sets the DNS name server used to resolve names (this is optional, since the router does not have to perform name resolution).

  3. Reconfigure the WRT54G hardware MAC address:

    Sometimes, when the Wireless AP is using MAC filtering, it may be necessary to change the hardwareMAC address of the WRT54G wirelss interface. This can be done using the following command:

    # nvram set il0macaddr=

    However, it is recommended to keep a copy of the original MAC address. My WRT54G router has a sticker in its bottom listing the hardware MAC address for the wired interfaces, but no sticker for the wireless one. Anyways, the MAC address for the wireless interface is the result of adding 0×02 to the last byte of the wired interface MAC address. Thus, if the wired MAC address is 00:11:22:33:44:55, the wireless MAC address is 00:11:22:33:44:57.

  4. Check the wireless interface is properly configured and working:

    The simplest way is to save changes into NVRAM and reboot to check everything is working is to save the changes to NVRAM and reboot:

    # nvram commit
    # reboot

    After rebooting, use the iwconfig command to check if the wireless interface has been able to associate with the Wireless AP:

    # iwconfig eth1
    eth1      IEEE 802.11-DS  ESSID: 
              Mode:Managed  Frequency:2.412Ghz  Access Point: AA:BB:CC:DD:EE:FF
              Tx-Power:31 dBm
              RTS thr=2347 B   Fragment thr=2346B
              Encryption key:XXXX-XXXX-XX

    where AA:BB:CC:DD:EE:FF is the MAC address of the Wireless AP.

  5. Harden the firewall:

    Since the WRT54G will be directly exposed to the outside by means of the Wireless connection, it is important to properly harden the firewall:

    # nvram set wan_ifname=eth1

    Although we are not using the WAN interface, the firewall initscript (S45firewall), by default, blocks all incoming traffic coming from the interface listed in the wan_ifname NVRAM variable. Since we want to block all the traffic coming from the Wireless interface, we want to block all traffic coming from eth1.

    Next, tweak the firewall initscript:

    # rm /etc/init.d/S45firewall
    # cp /rom/etc/init.d/S45firewall /etc/init.d/S45firewall
    # vi /etc/init.d/S45firewall

    From the /etc/init.d/S45firewall file, comment out the following lines:

    iptables -A INPUT -p icmp -j ACCEPT
    iptables -A INPUT -p gre -j ACCEPT
    iptables -A INPUT -p tcp -j REJECT --reject-with tcp-reset
    iptables -A INPUT -j REJECT --reject-with icmp-port-unreachable

    This will drop any incoming traffic not related to an already established flow (either TCP session, or UDP/ICMP datagrams).

Linksys WRT54G

September 6th, 2005

Today, I bought a Linksys WRT54G V3.1 Wireless Router from my usual local store. Although it uses a heavily customized Linux version internally, it is quited limited. For instance, it does not allow remote administration via SSH, only through a nice Web interface. So, after playing a little bit with it, I decided to install OpenWRT onto it. At the time of this writing, the only stable, binary release was WhiteRussian -RC2 from July, 19th.

Installing OpenWRT is a little bit tricky. Before being able to flash the OpenWRT firmware, you are instructed to set a NVRAM variable named boot_wait to on since this will introduce a three second delay, just after the router is powered up, for the administrator to upload a new firmware image via TFTP. Also, enabling boot_wait makes troubleshooting and recovery a little bit easier.

The problem is that changing this NVRAM variable is not easy, since the router’s Web interface does not offer that ability. Instead, as instructed by the OpenWRT documentation Wiki, I exploited a bug in the Ping.asp page of the Web interface which allows injecting shell-code to the router. Shortly, the shell code makes a copy of the file /usr/sbin/nvram into /tmp/n and then subsequently uses it to run the following commands:

/tmp/n set boot_wait=on
/tmp/n commit

The next step is launcing a TFTP client on a workstation attached to one of the four LAN ports of the WRT54G and have it prepared to upload the firmware binary image to the router at its factory-default address 192.168.1.1.

The OpenWRT download page offers firmware binary images for several hardware platforms in two formats:

  1. SquashFS:

    SquashFS is the preferred one, since it’s the most mature and allows for an easier recovery in case the routers filesystem gets corrupted.

    The SquashFS firmware is composed of two parts: the combination of a SquashFS ROM filesystem and a JFFS2 writeable flash filesystem. The SquashFS ROM is mounted at /root while JFFS2 is used as / containing a lot of symbolic links to files located under /root (that is, in the ROM).

  2. JFFS2:

    JFFS2 uses JFFS2 filesystem entirely. Thus, the whole filesystem is writeable, which can lead to accidental corruption. Recovery is a more difficult, since critical configuration or binary files could get destroyed or corrupted.

So, I chose the SquashFS firmware. I fired up my TFTP client and configured it to auto-retransmit the firmware file continuously, for 60 seconds, to the router’s IP (which, by default, is 192.168.1.1):

tftp 192.168.1.1
tftp> binary
tftp> rexmt 1
tftp> timeout 60
tftp> trace
Packet tracing on
tftp> put openwrt-wrt54g-squashfs.bin

Once the router is powered up, it will wait for three seconds for the firmware to be uploaded, so there is a small time window for this to succeed. In fact, it took me twice to get the firmware downloaded properly to the WRT54G. The router rebooted itself, set the DMZ led on for a while, then set it off, which means the router is ready. Once the firmware was uploaded, the router rebooted itself into OpenWRT, a pure Linux box.

OpenWRT gives so much freedom when compared to the original firmware. By default, it bridges wired LAN and wireless traffic together, while still having a WAN interface, but it can be easily reconfigured to act as a three-leg router acting as Wireless AP, or it can be reconfigured so the Wireless interface behaves as a client instead.

I’m still playing with it, but I have a lot of expectations about it.

In Mac OS X Tiger, the Login Window will now show most users, even if the UID is below 500, unless the account is UID 0 (root) or user whose default shell is set to /dev/null.

Tiger’s login window preferences now supports a new setting called HiddenUsersList, which is an array of one or more user’s short account name. To hide a list of accounts from the Login Window, run the following command:

# defaults write /Library/Preferences/com.apple.loginwindow \
  HiddenUsersList -array-add account1 account2 account3 ...

where account1, account2, account3, … are the short user names for the accounts you want to hide. This would add additional names to any that might already exist.

To unhide all the names you may have previously hidden, run:

# defaults delete /Library/Preferences/com.apple.loginwindow \
  HiddenUsersList

As always, make a backup of any preference file you might change so you can move back to it.