Using Proximity Operators with PQF

The proximity operator @prox is a special and more restrictive version of the conjunction operator @and. Its semantics are described in section 3.7.2 (Proximity) of Z39.50 the standard itself, which can be read on-line at http://lcweb.loc.gov/z3950/agency/markup/09.html

In PQF, the proximity operation is represented by a sequence of the form

@prox exclusion distance ordered relation which-code unit-code
	

• exclusion. 0 = false (i.e. the proximity condition specified by the remaining parameters must be satisfied) or 1 = true (the proximity condition specified by the remaining parameters must not be satisifed).

• distance. An integer specifying the difference between the locations of the operands: e.g. two adjacent words would have distance=1 since their locations differ by one unit.

• ordered. 1 = ordered (the operands must occur in the order the query specifies them) or 0 = unordered (they may appear in either order).

• relation. Recognised values are 1 (lessThan), 2 (lessThanOrEqual), 3 (equal), 4 (greaterThanOrEqual), 5 (greaterThan) and 6 (notEqual).

• which-code. known or k (the unit-code parameter is taken from the well-known list of alternatives described in below) or private or p (the unit-code paramater has semantics specific to an out-of-band agreement such as a profile).

• unit-code. If the which-code parameter is known then the recognised values are 1 (character), 2 (word), 3 (sentence), 4 (paragraph), 5 (section), 6 (chapter), 7 (document), 8 (element), 9 (subelement), 10 (elementType) and 11 (byte). If which-code is private then the acceptable values are determined by the profile.

PQF queries

	dylan
	"bob dylan"
       

	@or "dylan" "zimmerman"
	@and @or dylan zimmerman when
	@and when @or dylan zimmerman
       

	@set Result-1
	@and @set seta setb
       

	@attr 1=4 computer
	@attr 1=4 @attr 4=1 "self portrait"
	@attrset exp1 @attr 1=1 CategoryList
	@attr gils 1=2008 Copenhagen
	@attr 1=/book/title computer
       

	@prox 0 3 1 2 k 2 dylan zimmerman
       

	@term string "a UTF-8 string, maybe?"
       

	@or @and bob dylan @set Result-1
	
	@attr 4=1 @and @attr 1=1 "bob dylan" @attr 1=4 "slow train coming"
	
	@and @attr 2=4 @attr gils 1=2038 -114 @attr 2=2 @attr gils 1=2039 -109
      

CCL Syntax

The CCL parser obeys the following grammar for the FIND argument. The syntax is annotated by in the lines prefixed by ‐‐.

      CCL-Find ::= CCL-Find Op Elements
                | Elements.

      Op ::= "and" | "or" | "not"
      -- The above means that Elements are separated by boolean operators.

      Elements ::= '(' CCL-Find ')'
                | Set
                | Terms
                | Qualifiers Relation Terms
                | Qualifiers Relation '(' CCL-Find ')'
                | Qualifiers '=' string '-' string
      -- Elements is either a recursive definition, a result set reference, a
      -- list of terms, qualifiers followed by terms, qualifiers followed
      -- by a recursive definition or qualifiers in a range (lower - upper).

      Set ::= 'set' = string
      -- Reference to a result set

      Terms ::= Terms Prox Term
             | Term
      -- Proximity of terms.

      Term ::= Term string
            | string
      -- This basically means that a term may include a blank

      Qualifiers ::= Qualifiers ',' string
                  | string
      -- Qualifiers is a list of strings separated by comma

      Relation ::= '=' | '>=' | '<=' | '<>' | '>' | '<'
      -- Relational operators. This really doesn't follow the ISO8777
      -- standard.

      Prox ::= '%' | '!'
      -- Proximity operator

     

       dylan
       
       "bob dylan"
       
       dylan or zimmerman
       
       set=1
       
       (dylan and bob) or set=1
       
      

       ti=self portrait
       
       au=(bob dylan and slow train coming)

       date>1980 and (ti=((self portrait)))
       
      

CCL Qualifiers

Qualifiers are used to direct the search to a particular searchable index, such as title (ti) and author indexes (au). The CCL standard itself doesn't specify a particular set of qualifiers, but it does suggest a few short-hand notations. You can customize the CCL parser to support a particular set of qualifiers to reflect the current target profile. Traditionally, a qualifier would map to a particular use-attribute within the BIB-1 attribute set. It is also possible to set other attributes, such as the structure attribute.

A CCL profile is a set of predefined CCL qualifiers that may be read from a file or set in the CCL API. The YAZ client reads its CCL qualifiers from a file named default.bib. There are four types of lines in a CCL profile: qualifier specification, qualifier alias, comments and directives.

	ti       u=4 s=1
	au       u=1 s=1
	term     s=105
	ranked   r=102
	date     u=30 r=o
      

	 ti,ranked=knuth computer
	

	 date > 1980
	

	 ti > 1980
	

CCL API

All public definitions can be found in the header file ccl.h. A profile identifier is of type CCL_bibset. A profile must be created with the call to the function ccl_qual_mk which returns a profile handle of type CCL_bibset.

To read a file containing qualifier definitions the function ccl_qual_file may be convenient. This function takes an already opened FILE handle pointer as argument along with a CCL_bibset handle.

To parse a simple string with a FIND query use the function

struct ccl_rpn_node *ccl_find_str (CCL_bibset bibset, const char *str,
                                   int *error, int *pos);
     

which takes the CCL profile (bibset) and query (str) as input. Upon successful completion the RPN tree is returned. If an error occur, such as a syntax error, the integer pointed to by error holds the error code and pos holds the offset inside query string in which the parsing failed.

An English representation of the error may be obtained by calling the ccl_err_msg function. The error codes are listed in ccl.h.

To convert the CCL RPN tree (type struct ccl_rpn_node *) to the Z_RPNQuery of YAZ the function ccl_rpn_query must be used. This function which is part of YAZ is implemented in yaz-ccl.c. After calling this function the CCL RPN tree is probably no longer needed. The ccl_rpn_delete destroys the CCL RPN tree.

A CCL profile may be destroyed by calling the ccl_qual_rm function.

The token names for the CCL operators may be changed by setting the globals (all type char *) ccl_token_and, ccl_token_or, ccl_token_not and ccl_token_set. An operator may have aliases, i.e. there may be more than one name for the operator. To do this, separate each alias with a space character.

CQL parsing

A CQL parser is represented by the CQL_parser handle. Its contents should be considered YAZ internal (private).

#include <yaz/cql.h>

typedef struct cql_parser *CQL_parser;

CQL_parser cql_parser_create(void);
void cql_parser_destroy(CQL_parser cp);
      

To parse a CQL query string, the following function is provided:

int cql_parser_string(CQL_parser cp, const char *str);
      

int cql_parser_stream(CQL_parser cp,
                      int (*getbyte)(void *client_data),
                      void (*ungetbyte)(int b, void *client_data),
                      void *client_data);

int cql_parser_stdio(CQL_parser cp, FILE *f);
      

CQL tree

The the query string is valid, the CQL parser generates a tree representing the structure of the CQL query.

struct cql_node *cql_parser_result(CQL_parser cp);
      

Each node in a CQL tree is represented by a struct cql_node. It is defined as follows:

#define CQL_NODE_ST 1
#define CQL_NODE_BOOL 2
struct cql_node {
    int which;
    union {
        struct {
            char *index;
	    char *index_uri;
            char *term;
            char *relation;
	    char *relation_uri;
            struct cql_node *modifiers;
        } st;
        struct {
            char *value;
            struct cql_node *left;
            struct cql_node *right;
            struct cql_node *modifiers;
        } boolean;
    } u;
};
      

The search term node has five members:

• index: index for search term. If an index is unspecified for a search term, index will be NULL.

  index_uri: index URi for search term or NULL if none could be resolved for the index.

• term: the search term itself.

• relation: relation for search term.

• relation_uri: relation URI for search term.

• modifiers: relation modifiers for search term. The modifiers list itself of cql_nodes each of type ST.

The boolean node represents both and, or, not as well as proximity.

• left and right: left - and right operand respectively.

• modifiers: proximity arguments.

CQL to PQF conversion

Conversion to PQF (and Z39.50 RPN) is tricky by the fact that the resulting RPN depends on the Z39.50 target capabilities (combinations of supported attributes). In addition, the CQL and SRW operates on index prefixes (URI or strings), whereas the RPN uses Object Identifiers for attribute sets.

The CQL library of YAZ defines a cql_transform_t type. It represents a particular mapping between CQL and RPN. This handle is created and destroyed by the functions:

cql_transform_t cql_transform_open_FILE (FILE *f);
cql_transform_t cql_transform_open_fname(const char *fname);
void cql_transform_close(cql_transform_t ct);
      

The handle is destroyed by cql_transform_close in which case no further reference of the handle is allowed.

When a cql_transform_t handle has been created you can convert to RPN.

int cql_transform_buf(cql_transform_t ct,
                      struct cql_node *cn, char *out, int max);
      

If conversion failed, cql_transform_buf returns a non-zero SRW error code; otherwise zero is returned (conversion successful). The meanings of the numeric error codes are listed in the SRW specifications at http://www.loc.gov/srw/diagnostic-list.html

If conversion fails, more information can be obtained by calling

int cql_transform_error(cql_transform_t ct, char **addinfop);
      

The SRW error-codes may be translated into brief human-readable error messages using

const char *cql_strerror(int code);
      

If you wish to be able to produce a PQF result in a different way, there are two alternatives.

void cql_transform_pr(cql_transform_t ct,
                      struct cql_node *cn,
                      void (*pr)(const char *buf, void *client_data),
                      void *client_data);

int cql_transform_FILE(cql_transform_t ct,
                       struct cql_node *cn, FILE *f);
      

Specification of CQL to RPN mapping

The file supplied to functions cql_transform_open_FILE, cql_transform_open_fname follows a structure found in many Unix utilities. It consists of mapping specifications - one per line. Lines starting with # are ignored (comments).

Each line is of the form

       CQL pattern =   RPN equivalent
      

An RPN pattern is a simple attribute list. Each attribute pair takes the form:

       [set] type=value
      

The following CQL patterns are recognized:

       set.cql    = http://www.loc.gov/zing/cql/context-sets/cql/v1.1/
       set.dc     = http://www.loc.gov/zing/cql/dc-indexes/v1.0/

       index.cql.serverChoice = 1=1016
       index.dc.title         = 1=4
       index.dc.subject       = 1=21
  
       relation.<                 = 2=1
       relation.eq                = 2=3
       relation.scr               = 2=3

       position.any               = 3=3 6=1

       structure.*                = 4=1

      

        computer
       

        @attr 1=1016 @attr 2=3 @attr 4=1 @attr 3=3 @attr 6=1 "computer"
       

        computer^
       

        >my = "http://www.loc.gov/zing/cql/dc-indexes/v1.0/" my.title = x
       

        @attr 1=4 @attr 2=3 @attr 4=1 @attr 3=3 @attr 6=1 "x"
       

CQL to XCQL conversion

Conversion from CQL to XCQL is trivial and does not require a mapping to be defined. There three functions to choose from depending on the way you wish to store the resulting output (XML buffer containing XCQL).

int cql_to_xml_buf(struct cql_node *cn, char *out, int max);
void cql_to_xml(struct cql_node *cn, 
                void (*pr)(const char *buf, void *client_data),
                void *client_data);
void cql_to_xml_stdio(struct cql_node *cn, FILE *f);
      

cql_to_xml writes the result in a user defined output stream. cql_to_xml_stdio writes to a a file.