Analog 6.0: Search arguments

Sometimes a URL contains arguments after a question mark. For example, the URL

/cgi-bin/script.pl?x=1&y=2

runs the /cgi-bin/script.pl program with arguments x=1 and y=2. (Sometimes the server records these arguments in a separate field in the logfile, but if so you can use the %q field in the LOGFORMAT command, and analog will translate the filename to the above format).

You can tell analog either to read or to ignore the arguments using the commands ARGSINCLUDE and ARGSEXCLUDE which we'll discuss in a minute. But by default, all arguments are read, and as this is usually what you want, you don't usually need those commands.

You don't always see the arguments in the reports, even if they're being read, because analog doesn't show them if there aren't enough of them. In order to see them, you have to set the corresponding ARGSFLOOR parameter low enough.

Also note that within a report, the search arguments are listed immediately under the file to which they refer. This temporarily interrupts the normal order of the files. It may be clearer if you turn the N column on.

Assuming that the arguments are being read, analog treats the file /cgi-bin/script.pl?x=1&y=2 as a different file from /cgi-bin/script.pl (or from /cgi-bin/script.pl?y=2&x=1 for that matter). It doesn't look like that in the Request Report because you see a grand total for /cgi-bin/script.pl with all its different arguments. But it matters if you want to do inclusions and exclusions or aliases on the file.

The reason is that, for example, the command

FILEINCLUDE /cgi-bin/script.pl

doesn't match the file /cgi-bin/script.pl?x=1&y=2. To match that, you would have to use something like

FILEINCLUDE /cgi-bin/script.pl*

instead. Similarly

FILEALIAS /cgi-bin/script.pl /script.pl

will change /cgi-bin/script.pl itself, but not /cgi-bin/script.pl?x=1&y=2. You might want to use something like

FILEALIAS /cgi-bin/script.pl?* /script.pl?$1

as well. (However, PAGEINCLUDE and PAGEEXCLUDE always refer to the part of the filename before the question mark.)

Conversely, because in the Request Report files with arguments are only included if their parent file is included, you can't just

REQINCLUDE /cgi-bin/script.pl?*x=1*

or you will end up with nothing listed. You have to

REQINCLUDE /cgi-bin/script.pl

as well.

The alternative is to tell analog not to read the search arguments. There are commands called ARGSINCLUDE and ARGSEXCLUDE, and REFARGSINCLUDE and REFARGSEXCLUDE, to do this. They work the same as the other INCLUDE and EXCLUDE commands which we discussed in the previous section. So, for example, if the command

ARGSEXCLUDE /cgi-bin/script.pl

were given, analog would ignore the arguments to that file, and so read /cgi-bin/script.pl?x=1&y=2 as just /cgi-bin/script.pl. On the other hand, if

ARGSINCLUDE /cgi-bin/script.pl

were specified, analog would read the arguments, and so treat /cgi-bin/script.pl?x=1&y=2 as a different file from /cgi-bin/script.pl. REFARGSINCLUDE and REFARGSEXCLUDE are the same for referrers.

Technical note: the check for whether the arguments should be included happens before the filename has been subject to either built-in or user-specified aliases. So you have to use the unaliased name, exactly as it occurs in the logfile. For example, ARGSINCLUDE /~sret1/script.pl won't match /%7Esret1/script.pl even though they are really the same file. It also means that you can't use "pages" in the ARGSINCLUDE or ARGSEXCLUDE command, because we don't know whether a file is a page until after it's been aliased.

There are related commands called SEARCHENGINE and INTSEARCHENGINE. If you have referrers with search arguments, usually from search engines, you can tell analog which field corresponds to the search term. It uses this information to compile the Search Query Report and the Search Word Report. For example, consider the referrer

http://www.altavista.com/cgi-bin/query?pg=q&kl=XX&q=carrot+cake

The search term is in the field q= so the appropriate SEARCHENGINE command is

SEARCHENGINE http://www.altavista.com/cgi-bin/query q

(or even better

SEARCHENGINE http://*altavista.*/* q

to allow for all their mirror sites in different countries.)

The command INTSEARCHENGINE is the same for search engines, or other scripts which take arguments, within your site. For example, you might have requests for files like

/cgi-bin/search?trm=chocolate+cake

in which case you would specify

INTSEARCHENGINE /cgi-bin/search trm

and (assuming you haven't done an ARGSEXCLUDE for that file) "chocolate cake" would then appear in your Internal Search Query Report.

Sometimes a search engine has two or more possible fields for the search term. In that case you can list all of them separated by commas, like this:

SEARCHENGINE http://*webcrawler.*/* search,searchText

The rest of this section is a bit technical, and you usually don't need to worry about it. On a first reading, you probably want to skip it.

I said previously that %7E in a URL is automatically converted to ~, etc. In fact this is only done to the ASCII-printable characters %20-%7E, because these are the only characters that are the same in every character set. (In fact, even that isn't true. Experts might want to know that ?, &, ; and = aren't converted either, to distinguish them from query-string delimiters: an encoded ?, &, ; or = is one that is not intended to be a delimiter. Also % isn't converted, to avoid confusing %25nm with %nm.)

But in the Search Query Report and Search Word Report it is useful to be able to convert non-ASCII characters too, so that you can see the actual words people typed, rather than get the %nm codes in place of all accented letters. So in these reports analog also converts characters %A0-%FF (if you are using an ISO-8859-* character set) or %80-%FF (for most other character sets).

However, there are reasons why you might not want this feature, and you can turn it off with the command

SEARCHCHARCONVERT OFF

These reasons include:

The character set in which the query was submitted to the search engine may not be the same as that in which the page reached was written, or that in which the analog output page is being written. So converting to the character set of the analog output page may give garbage anyway. This is particularly a problem with languages, such as Russian, which have two or more characters sets in common use. It is also a problem for sites which host resources in many languages.
Not all of the character positions correspond to printable characters in every character set. Analog knows that %80-%9F are non-printable in the ISO-8859-* character sets, but apart from that it converts everything in %80-%FF. So you may end up with non-printable characters in your output.

SEARCHCHARCONVERT is always turned off if the output is in ASCII; and it defaults to off if the output is in a multibyte character set because it doesn't work well in that case.

Go to the analog home page.

Stephen Turner
19 December 2004

Need help with analog? Use the analog-help mailing list.

[ Top | Up | Prev | Next | Map | Index ]