Chapter 4. Upgrading from Netatalk 1.x

Jörg Lenneis

initial version 

26 June, 2004

Table of Contents

Overview
Volumes and filenames
How to upgrade a volume to 2.x
How to use a 1.x CAP encoded volume with 2.x
How to use a 1.x NLS volume with 2.x
Choosing a CNID storage scheme
How to upgrade if no persistent CNID storage was used
How to upgrade if a persistent CNID storage scheme was used
How to upgrade if a persistent CNID storage scheme was used, the brute force approach
Setting up a test server on the same machine
Setting up an empty test share
Duplicating an already existing share
Configuring and running the test afpd

Overview

Version 2.x of the Netatalk suite includes significant changes and enhancements in functionality compared to 1.x. Netatalk 2.x is now supported which allows UTF8-MAC encoded filenames of up to 255 bytes (85-255 chars) in length amongst other things. The Catalogue Node ID (CNID) subsystem has been reworked as well and should now be much more robust. For an overview of what CNIDs are and why you need them please see the CNID section in the manual.

The downside of these enhancements is that upgrading to Netatalk 2.x is not a process that can be easily automated. Too many factors depend on site specific configuration and administrators have to make choices that suit their requirements. This document attempts to clarify the issues and outline the steps that need to be taken for a successful upgrade. As usual, the first of these steps should be to make a complete backup of all volumes and home directories that were in use with Netatalk before. Afterwards, you'll have to decide

  1. what encoding to use for filenames in the future and how to convert existing filenames

  2. what storage scheme to use for CNIDs and maybe convert an existing database to that scheme

The following two sections deal with each of these areas in turn.

Volumes and filenames

Netatalk 1.x saved filenames in the so called CAP encoding by default. Alternatively, there was the NLS system, that allowed you to convert filenames to other codepages, like ISO-8859-1.

For Netatalk 2.x the charset conversion routines had to be completely rewritten to support AFP 3.x. For more indepth information on character sets please read the Unicode/charsets section in the manual.

As a consequence, Netatalk 2.x now stores filenames in UTF-8 by default. Additionally you have to specify a maccodepage in afpd.conf, if your Mac clients are not using MacRoman.

The format of the metadata files stored in the .AppleDouble folders has changed from AppleDouble v1 to AppleDouble v2. Netatalk 2.x is still able to use AD1 files, if configured. Otherwise ADv1 files will silently be updated to the new ADv2 format, which will prevent you from using this volume with 1.x again.

Warning

Do not share a 1.x volume with Netatalk 2.x without setting the proper options!

Warning

In case of MacChineseTraditional, MacJapanese or MacKorean, This method cannot be used.

Before upgrading, copy all files to the client. And restore all files to netatalk 2.0.4 or later.

Note

You should consider 'upgrading' your volumes using the new defaults UTF-8 and AppleDouble v2, even if this is a time consuming process. AFP 3.x uses UTF-8 and it is impossible to fully map UTF-8 to any of the old volume formats.

How to upgrade a volume to 2.x

To convert the 1.x CAP or NLS encoded volumes on the server, we provide the uniconv(1) utility. Please see the man page for details.

Another option to perform an upgrade, is to copy all files using a Mac client. Either copy the volume to a Mac while you are still running 1.6, then install 2.x and copy the data back to a fresh share, or try to set up the volume with the compatibility options described below and do a share to share copy.

How to use a 1.x CAP encoded volume with 2.x

Using a 1.x CAP encoded volume is still possible with Netatalk 2.x. To work properly, the following options need to be set, matching your 1.x setup:

afpd.conf:

  • maccodepage

AppleVolumes.default:

  • volcharset

  • adouble

You have to make sure maccodepage matches your Apple clients codepage. For western users the default Mac_Roman should be fine.

Set volcharset to ASCII.

Set adouble:v1, this will make sure the metadata files will not be changed to AppleDouble v2. If you do not set this option, it will not be possible to use the volume with Netatalk 1.x anymore.

Example:

afpd.conf:

 - -transall -maccodepage:MAC_CENTRALEUROPE

AppleVolumes.default:

 /path/to/share "1.x Volume" adouble:v1 volcharset:ASCII

How to use a 1.x NLS volume with 2.x

Whether you can still use an 1.x NLS encoded volume with Netatalk 2.x mainly depends on which NLS setting you used with 1.x.

Make sure you set the correct maccodepage in afpd.conf !

maccode.iso8859-1

Use the following settings in AppleVolumes.default:

 /path/to/share "1.x Volume" adouble:v1 volcharset:ISO-8859-1
maccode.iso8859-1.adapted

Sorry, you're out of luck. This NLS contains a non standard mapping and is not supported by afpd anymore. You'll have to convert the volume to a supported encoding.

maccode.437

Using the following settings in AppleVolumes.default might work, but is untested:

 /path/to/share "1.x Volume" adouble:v1 volcharset:CP437
maccode.850

Using the following settings in AppleVolumes.default might work, but is untested:

 /path/to/share "1.x Volume" adouble:v1 volcharset:CP850
maccode.koi8-r

Using the following settings in AppleVolumes.default might work, but is untested:

 /path/to/share "1.x Volume" adouble:v1 volcharset:KOI8-R

Note

All of the above require iconv to be installed and to supply the volcharset codepage!

Choosing a CNID storage scheme

Previous versions of Netatalk allocated CNIDs either on the fly or CNIDs were recorded in a persistent database. "On the fly methods" work by either generating a CNID from the device and inode number or simply by using a counter that is increased by one on each access to a file or directory from the client. The counter only lasts for the lifetime of an afpd daemon process and inode numbers are reused for a different file once the original file has been deleted. These methods therefore violate a fundamental assumption: A CNID once assigned must never be reused for the lifetime of a volume. Netatalk 2.0 supports one "On the fly scheme" called last. It computes CNIDs for files from device and inode of the file and uses a counter for directories. You should think twice about using it in production. Depending on your needs and the semantics of the underlying file system it might be OK on read only volumes, but even there we are not certain if OS X clients will work properly.

That leaves the CNID schemes that use persistent storage for CNIDs. Netatalk 2.3 supports one: dbd. It is based on the Berkeley DB database library as before. One difference is, though, that you are not restricted to using a single scheme for all of your volumes that has to be determined at compile time. The CNID scheme (also called a "CNID backend") is now a runtime option for a volume. That means that you can make the choice per volume based on your requirements. Here are the properties as well as the advantages and disadvantages of the two supported schemes:

  1. last: See above. Avoid, if at all possible.

  2. dbd: There is only a single daemon that accesses the CNID database for a given volume. Any afpd process that wishes to retrieve or update CNIDs for that volume needs to do it via the daemon. The CNID can database be (this is a compile time option) updated under Berkeley DB transactional protection. This design combined with the transactional updates makes the CNID database crashproof: Any of the participating afpd daemons, the database daemon itself or the whole machine can crash and the CNID database should still be in a consistent state. Also, have a look at the nosync option documented in the cnid_dbd manual page.

Note that the dbd backend needs an auxiliary daemon, called cnid_metad, to work. It should be started together with afpd. If the dbd backend is compiled into afpd (the default), this should happen automatically. If you cannot find it in the process list even though the dbd backend is used please check for errors in the startup scripts.

How to upgrade if no persistent CNID storage was used

That is easy. Just pick a CNID backend from above, configure it properly in afpd.conf and the AppleVolumes file and start up the necessary Netatalk processes. The databases will be automatically created in a subdirectory .AppleDB of the volume in question.

How to upgrade if a persistent CNID storage scheme was used

In that case the CNID databases need to be upgraded. A script called cnid2_create that comes with Netatalk 2.0 does most of the work. The steps you have to take depend on what version of Berkeley DB is installed on your system. If you already use one of the supported versions of Berkeley DB (4.1.25 or 4.2.52) for your old Netatalk installation and plan to use it for Netatalk 2.0 as well just use the db_dump and db_load utilities that came with it as indicated below. Otherwise it is probably best to have the old and the new (to be used with Netatalk 2.0) version of Berkeley DB installed side by side until you have finished the upgrade. The reason for this is that we will dump out the old databases with the currently installed version of Berkeley DB in ASCII format and reload them with the new version. This avoids any incompatibility problems between Berkeley DB releases with respect to the on-disk format.

For each volume to be upgraded, follow these steps

  • Stop all afpd daemons accessing the volume.

  • Change to the database directory for that volume, most likely the .AppleDB subdirectory of the volume toplevel directory in question.

  • Dump the contents of cnid.db and didname.db using the old (installed) version of Berkeley DB like this:

     db_dump -f cnid.dump cnid.db 
     db_dump -f didname.dump didname.db

    Make sure the db_dump utility you are using is the correct (currently used) one. Use the full path to the db_dump executable if in doubt. So if this database was created with Berkeley DB 3.xx installed in /usr/local/db3 use /usr/local/db3/bin/db_dump instead. This will create two files, cnid.dump and didname.dump in the current directory.

  • Run the cnid2_create script:

     /path/to/netatalk/bin/cnid2_create 

    The script assumes that .AppleDB is a subdirectory of the volume directory to be upgraded. If that is not the case give the full path name of the volume as the first argument to cnid2_create. The script will create a file cnid2.dump in ASCII format.

  • Remove the old Berkeley DB environment and logfiles (if present):

     rm __db.* log.*
  • Load cnid2.dump into the new database. You should use the db_load utility of Berkeley DB that will be used with version 2.0 of Netatalk. So if Berkeley DB 4.xx lives in /usr/local/db4 use

     /usr/local/db4/bin/db_load -f cnid2.dump cnid2.db 

    This will create the new database file, cnid2.db. Remove the old database files cnid.db, didname.db and devino.db. The intermediate files cnid.dump, didname.dump and cnid2.dump can be removed now or at some later time.

If you do not want to have two versions of Berkeley DB installed side by side during the upgrade, you should first dump out the old databases as indicated above for all volumes, upgrade Berkeley DB and then load them with db_load. The cnid2_create script can be run before or after the upgrade. Berkeley DB environment and logfiles should still be removed before running db_load.

How to upgrade if a persistent CNID storage scheme was used, the brute force approach

If you are absolutely sure what you are doing, you can also just clear out all database files from the .AppleDB directories. They will be recreated, but will not contain the same CNIDs as before!! That might lead to all sorts of problems, like aliases not working any more on clients. As I said, make sure you know the consequences and don't mind them.

Setting up a test server on the same machine

Providing a test environment in parallel with the existing production installation is not difficult and, if done properly, it should not in any way disrupt the normal operation. However, as always, it is recommended to make a backup of the existing installation before proceeding. When compiling a newer netatalk version you should also take care that you do not overwrite the binaries of an older version (make use of the --prefix= configure option).

There could be more than two afpd servers running on one UNIX box. You just have to be careful to keep them from running into each other:

  • the shares/volumes (AppleVolumes.default)

  • the PID file (afpd -P command line option)

  • the port number (-port option in afpd.conf)

  • no use of AppleTalk (-noddp option in afpd.conf)

You should test the new Netatalk version with both a freshly created new share and another one that has been duplicated/converted from an already existing volume. This helps finding mistakes you probably made in the upgrade process when the first share behaves well and the latter not.

Setting up an empty test share

First, you have to provide some space for the test share. Just create a directory on one of your data filesystems. However, this directory must not be accessible from the production afpd server. Don't forget to set appropriate permissions for the share. For example:

 mkdir /macdata/testshare
 chown root.macusers /macdata/testshare
 chmod g+wrx,g+s /macdata/testshare

Duplicating an already existing share

Ensure that users cannot access the share in question and copy the whole contents (including all the metadata directories like .AppleDB) to another location.

 cp -pr /production/testshare /macdata/

Then do the somewhat extensive upgrade of CNID databases and filename encodings outlined earlier in this chapter.

Configuring and running the test afpd

Normally the test afpd cannot listen on the standard afpovertcp port, because that one is already bound by the production afpd. So the afpd.conf should look like

 - -noddp -nouservol -port 5480 -loginmsg "WARNING: test server"

You also have to present the share to the Mac users by editing the AppleVolumes.default file. Remove the line containing a single "~" at the end of the file and append something like:

 /macdata/testshare "Test Volume (not production)"

That should be the only uncommented line in the file. Do not forget to adjust encoding and AppleDouble setting when you're not using the recommended defaults UTF8 and ADv2.

In case you have many users and want to restrict access to the test server, there is a provision for that in the AppleVolumes.default file. First, create a group named, say, afpdtest, and put in it all users you would want to enable access to the test volumes. Then, instead of the line above, append a line like this one:

 /macdata/testshare "Test Volume (not production)" allow:@afpdtest

The test afpd server can be started now:

 $TESTDIR/sbin/afpd -P /var/run/afpd-test.pid

You can also put this line in the production netatalk start script, in the "start" case. In the "stop" case, you should insert

 [ -f /var/run/afpd-test.pid ] && kill `cat /var/run/afpd-test.pid`

The production server has to know about the test server. Otherwise the Mac users would not be able to see the test server in their choosers. Append the following line to the production afpd.conf (usually in /etc/netatalk/afpd.conf):

 "Test server (not production)" -proxy -uamlist "" -port 5480

and restart the production netatalk. Note that the port directive here should match the one which appears above in the test afpd.conf.

Important

Note that there is a limit of 31 characters for the server's name. Should the name be longer, then afpd will just refuse to register the server.

The test server should appear in the chooser on Macs. You can also test that from the UNIX command line:

 nbplkup =:AFPServer

or

 netstat -an | grep 5480

If everything went fine spread the word about the test server among your more experienced Mac users and see whether things work as expected.