Analog 6.0: Coping with low memory

This section describes how to run analog with lower amounts of memory. For a normal logfile this will make analog run a bit slower. But if your computer is running out of memory when running analog, it will go very slowly indeed: so for large logfiles, this can make analog run much faster, or even make an analysis possible that wouldn't otherwise be possible.

Recall what happens to an item when it has been read in. First it is aliased. Secondly, it is checked to see whether it is included or excluded. Then finally, if all the items are wanted, one request is added to its score.

Normally the name of the item is saved before the aliasing takes place. This avoids analog having to do the aliasing again next time the same item is encountered. But this can take up more memory than necessary. So there is a family of LOWMEM commands provided, which tell analog to record the name at a later stage, or even not at all. If you use these commands, analog will have to do a bit more work than normal, but it will use less memory. On most sites, the hosts take up most of the memory, so I'll use the HOSTLOWMEM command as an example.

The command

HOSTLOWMEM 0

represents the normal case, when the hostname is recorded before being aliased. If you specify

HOSTLOWMEM 1

instead, then the hostname is not recorded until after the aliasing. If you specify

HOSTLOWMEM 2

then the name is not recorded until after the inclusion and exclusion lookup has been done as well. And finally, if you give the command

HOSTLOWMEM 3

then the hostname is not saved at all, and the Host Report will not be constructed, even if you've asked for it. (The Domain Report can still be constructed though.) The analogous commands for the other items are FILELOWMEM, BROWLOWMEM, REFLOWMEM, USERLOWMEM and VHOSTLOWMEM.

So what should you do if analog runs out of memory? First, look in your logfile to see which items are taking up all the memory. If you have lots of different filenames, ones you generate on the fly for example, you would want to use the FILELOWMEM commands. Maybe you could combine all the similar filenames into one with a FILEALIAS command, and use FILELOWMEM 1. (If you have lots of different filenames caused by different search arguments, then using ARGSEXCLUDE might solve your problem without any need to use LOWMEM at all). But for most users, it is the hostnames which cause the problem. If you only want to analyse requests from certain hosts, then you could use HOSTLOWMEM 2 to exclude the others before recording those that are left. If you don't want to exclude any hosts, and you haven't got enough memory to record all the different hostnames, then HOSTLOWMEM 3 would be appropriate.

Go to the analog home page.

Stephen Turner
19 December 2004

Need help with analog? Use the analog-help mailing list.

[ Top | Up | Prev | Next | Map | Index ]