Chapter 15. Fetching

This is the first paragraph of Fetching.

FETCHING AND UPDATING:

A convention about using the "update" and "fetch" targets makes it easy for
users to know how to use a recipe.  The main recipe for a project should be
able to be used in three ways:

1. Without specifying a target.
	This should build the program in the usual way.  Files with an
	"fetch" attribute are obtained when they are missing.
2. With the "fetch" target.
	This should obtain the latest version of all the files for the
	program, without building the program.
3. With the "update" target.
	This should fetch all the files for the program and then build it.
	It's like the previous two ways combined.

Here is an example of a recipe that works this way:

	STATUS = status.txt
	SOURCE = main.c version.c
	INCLUDE = common.h
	TARGET = myprog

	$TARGET : $SOURCE
		:cat $STATUS
		:sys $CC $CFLAGS $LDFLAGS -o $target $source
	$TARGET : $STATUS

	# specify where to fetch the files from
	:attr {fetch = cvs://:pserver:anonymous@cvs.myproject.sf.net:/cvsroot/myproject} $SOURCE $INCLUDE
	:attr {fetch = ftp://ftp.myproject.org/pub/%file%} $STATUS

When the "fetch" target is not specified in the recipe or its children, it
is automatically generated.  Its build commands will fetch all nodes with
the "fetch" attribute, except ones with a "constant" attribute set
(non-empty non-zero).  To do the same manually:

	fetch:
		:fetch $SOURCE $INCLUDE $STATUS

NOTE: When any child recipe defines a "fetch" target no automatic fetching
is done for any of the recipes.  This may not be what you expect.

When there is no "update" target it is automatically generated.  It will
invoke the "fetch" target and the first target in the recipe.  To do
something similar manually:

	update: fetch $TARGET

Although the automatically generated "update" target uses one of $TARGET,
"all" or the first encountered dependency.

:fetchall
:fetchall {attr = val}
		Fetch all the files in the recipe (and child recipes) that
		have the "fetch" attribute.
		Extra attributes for fetching can be specified here, they
		overrule the attributes of the file itself.

aap fetch		Does a ":fetchall".

	

THE FETCH ATTRIBUTE

The "fetch" attribute is used to specify a list of locations where the file
can be fetched from.  The word at the start defines the method used to
fetch the file:
	ftp		from ftp server
	http		from http (www) server
	scp		secure copy
	file		local file system
	cvs		from CVS repository
			For a module that was already checked out the part
			after "cvs://" may be empty, CVS will then use the
			same server (CVSROOT) as when the checkout was done.
	other 		user defined

These kinds of locations can be used:

	ftp://ftp.server.name/full/path/file
	http://www.server.name/path/file
	scp://host.name/path:path/file
	cvs://:METHOD:[[USER][:PASSWORD]@]HOSTNAME[:[PORT]]/path/to/repository
	file:~user/dir/file
	file:///etc/fstab

For a local file there are two possibilities: using "file://" or "file:".
They both have the same meaning.  Note that for an absolute path, relative to
the root of the file system, you use either one or three slashes, but not two.
Thus "file:/etc/fstab" and "file:///etc/fstab" are the file "/etc/fstab".  A
relative path has two or no slashes, but keep in mind that moving the recipe
will make it invalid.  You can also use "file:~/file" or "file://~/file" for a
file in your own home directory, and "file:~jan/file" or "file://~jan/file"
for a file in the home directory of user "jan".

To add a new method, define a Python function with the name "fetch_method",
where "method" is the word at the start.  The function will be called with two
arguments:
	name		the url with "method://" removed
	dict		a dictionary with all attributes used of the URL,
			dict["name"] is the full URL
	node		a node object.  Useful items:
			   node.name		short name
			   node.absname		full name of the file
			   node.recipe_dir	directory in which node.name
						is valid
			   node.attributes 	dictionary with attributes
The function should return a non-zero number for success, zero for failure.
Here is an example:

    :python
	def fetch_foo(from_name, dict, node):
	    to_name = node.absname
	    try:
		foo_the_file(from_name, to_name, dict["foo_option"])
	    except:
	    	return 0
	    return 1

In the "fetch" attribute the string "%file%" can be used where the path of
the local target is to be inserted.  This is useful when several files have a
common directory.  Similarly "%basename%" can be used when the last item in the
path is to be used.  This removes the path from the local file name, thus can
be used when the remote directory is called differently and only the file name
is the same.  Examples:

      :attr {fetch = ftp://ftp.foo.org/pub/foo/%file%} src/include/bar.h

Gets the file "src/include/bar.h" from
"ftp://ftp.foo.org/pub/foo/src/include/bar.h".

      :attr {fetch = ftp://ftp.foo.org/pub/foo/src-2.0/include%basename%}
			  src/include/bar.h

Gets the file "src/include/bar.h" from
"ftp://ftp.foo.org/pub/foo/src-2.0/include/bar.h".


CACHING:

Remote files are downloaded when used.  This can take quite a bit of time.
Therefore downloaded files are cached and only downloaded again when outdated.

The cache can be spread over several directories.  The list is specified
with the $CACHE variable.

NOTE: Using a global, writable directory makes it possible to share the cache
with other users, but only do this when you trust everybody who can login to
the system!

A cached file becomes outdated as specified with the "cache_update" attribute
or $cache_update.  The value is a number and a name.  Possible values for the
name:
	day	number specifies days
	hour	number specifies hours
	min	number specifies minutes
	sec	number specifies seconds
The default is "12 hour".

When a file becomes outdated, its timestamp is obtained.  When it differs
from when the file was last downloaded, the file is downloaded again.  When
the file changes but doesn't get a new timestamp this will not be noticed.

When fetching files the cached files are not used (but may be updated).