wxis is a module for working with CDS/ISIS databases from Python, via WXIS.

Alternative names: xispy, pywxis, wxispy.

Problema con el nombre del módulo Python: es muy cómodo llamarlo wxis (archivo wxis.py), pero en algunas situaciones se presta a confusión con wxis el programa.

README file for wxis


Requirements:

    * Server:
      - web server with CGI support enabled
      - wxis (5.x or higher)
    * Client:
      - Python (tested with version 2.5)

Files:

    wxis-json-modules/  (put it somewhere in your web server's cgi-bin folder)
        _common.xis
        _display-record.xis
        control.xis
        delete.xis
        edit.xis
        extract.xis
        index.xis
        list.xis
        search.xis
        write.xis
        
    wxis: copy it in the same folder as the .xis files (or use a symlink to a different location, always below your cgi-bin folder)

    wxis/  (put this folder wherever you please, on the client)
        wxis.py
        config.py
        test/       (test database files)
            cds.iso
            cds.mst
            cds.xrf
            cds.fst
        
Note: although you can test wxis on a standalone computer, you can also work with a separate database server, i.e. web server + databases + wxis + *.xis files living on a computer, and python + *.py files on another one. However, to make things easier, the ''test'' module assumes that the database is local.
        
Edit ''config.py'' and adjust a few parameters.

Check permissions for the web server user in the ''test'' folder. This user must be able to write and create files there.
        
Run the command

    python wxis.py

Compare results with ...

Versión: 2008-03-28

# coding=utf-8
 
"""
wxis
A module for accessing CDS/ISIS databases through Bireme's WXIS. 
 
MIT License <http://www.opensource.org/licenses/mit-license.php>
 
(c) 2008 Fernando J. Gómez / INMABB / Conicet
 
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
 
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
 
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
"""
 
 
def rename_key(oldkey, newkey, dict):
    """
    Replaces oldkey by newkey in dictionary dict.
    """
    try:
        dict[newkey] = dict[oldkey]
        del dict[oldkey]
    except KeyError:
        pass
    return dict
 
def remote_call(url, data):
    """
    Opens an URL and returns the response.
    TO-DO: move 'proxies' to a config file? Use 'proxies={}' to avoid looking for proxies when wxis is in localhost. 
    """
    from urllib import urlopen
    try:
        fp = urlopen(url, data, proxies={})  # NOTE: 'POST' is implied when a second positional param ('data') is used
    except IOError:
        return 'Error connecting to database server.'
        raise  # TO-DO: test this
    else:
        return fp.read()
 
# Should be defined inside the class IsisDb?        
def get_status(resp):
    """Returns the value of Isis_Status."""
    return resp['meta']['Isis_Status']
 
# Should be defined inside the class?
# Should be called automatically from an option in the constructor IsisDb.__init__?        
#def createdb(name):
#    """Creates a database."""
#    db = IsisDb(name)
#    db.control(function='create', create='master')
#    # If there was an error creating the database, it's been already handled by __doTask()
#    return db
 
 
class IsisDb:
    """
    Gives access to a CDS/ISIS database through WXIS.
 
    TO-DO: Besides the 'name' attribute, a database may have other associated attributes, such as
    fst, actab, uctab, stw (and maybe gizmo).
    """
 
    def __init__(self, name, **args):
        self.name = name
 
        # An optional keyword parameter 'create' means "create this db". Examples:
        #           books = IsisDb('/path/to/books')              # check master existence, raise exception if it does not exist  
        #           users = IsisDb('/path/to/users', create=True) # create unconditionally, don't check existence
        if 'create' in args.keys() and args['create'] == True:
            self.__create()
        elif not self.__exists():
            raise DatabaseDoesNotExist, self.name
 
    def __str__(self):
        return self.__class__.__name__ + ': ' + self.name 
 
    def __create(self):
        """
        Creates a database (only the master file).
        """
        self.control(function='create', create='master')
        # NOTE: If an error ocurrs while attempting to create the database,
        # it's handled by __doTask().
 
    def __exists(self):
        """
        Checks if master file exists.
        """
        resp = self.get_status() 
        if resp['database']['status']['master'] == 'not found':
            return False
        else:
            return True
 
    def __doTask(self, script, params, content=None):
        """
        This is the base method: builds an URL and the POST data, calls wxis, checks
        its response for errors, and finally returns the response or raises an
        exception.
 
        Parameters:
            script    Name of the IsisScript to invoke.
            params    Input parameters for the script. 
            content   (Optional) Record content to be written.
        """ 
 
        from urllib import urlencode
        import config
 
        # Build the URL
        url = 'http://%s:%s%s' % (config.HOST, config.PORT, config.PATH)
 
        # Append extra parameters. Note: all parameters are sent using POST
        IsisScript = '%s/%s.xis' % (config.SCRIPT_DIR, script)
        params.update({'IsisScript': IsisScript, 'database': self.name})
        if content:
            params.update({'content': content})
        data = urlencode(params)
 
        # Get WXIS's response
        wxis_response = remote_call(url, data)
        #print wxis_response
 
        # Now try to catch errors in the response
        try:
            # Try to create a Python object (a dictionary) from the response
            response = eval(wxis_response)
        except SyntaxError:
            # Reasons for a syntax error:
            #   (a) WXIS died:  "WXIS|some error|...|...|" 
            #       Some examples:
            #           WXIS|file error|file open|Isis_Script|
            #           WXIS|fatal error|unavoidable|dbxopen: /home/fernando/tmp/bibliox.xrf (2)|
            #           WXIS|execution error|invalid value|-1|
            #       For a comprehensive list of errors, see these semi-official docs:
            #           * http://ibama2.ibama.gov.br/cnia2/cisis/mensagens%20de%20erro%20do%20wxis-mx.pdf
            #           * http://www.elysio.com.br/documentacao/manual_phl81.pdf
            #           * http://www.google.com.ar/search?q=%22de+erro+do+CISIS%22&filter=0
            #
            #   (b) WXIS sent an ill-formed response (e.g. missing comma, mismatched brackets)
            #
            #   Errors of type (a) can be detected using a regular expression.
            import re
            pattern = re.compile(r'(WXIS\|.+ error\|.+$)')
            match = pattern.search(wxis_response)
            if match:
                raise WxisHardError, match.group()
            else:
                # This covers reason (b)
                raise BadResponseError, wxis_response
        else:
            # OK, so the response is clean JSON... but still we may have a (clean) error message
            try:
                # Did the script complain? 
                reason = response['error']
            except KeyError:
                # There's no 'error' key in the response -- return the Python object
                return response
            else:
                # We have an error of the 'soft' kind
                raise WxisSoftError, reason  
 
 
    # The following seven methods correspond to the original wxis-modules scripts
    # or basic functions.
    # NOTE: index.xis, list.xis and search.xis expect an optional 'from' parameter,
    # but since 'from' is a Python keyword, we use 'start' instead,
    # e.g. db.index(start='BAR', count=10)
 
    # TO-DO: rename method to mfnrange()? 
    def do_list(self, **params):
        """
        Retrieves a range of records.
 
        Parameters:
            start    (Optional)
            to       (Optional)
            count    (Optional)
        """
        params = rename_key('start', 'from', params)
        return self.__doTask('list', params)
 
    def search(self, **params):
        """
        Performs a search using the inverted file.
 
        Parameters:
            query        The search expression. Queries must use the CISIS search language,
                         which is based on the standard CDS-ISIS search language.
                         See http://www.ius.bg.ac.yu/biblioteka/isis_search.html
            start        (Optional)
            to           (Optional)
            count        (Optional)
            totalonly    (Optional) Use totalonly=1 to request the total number of results (no records)
        """
        params = rename_key('start', 'from', params)
        return self.__doTask('search', params)
 
    # TO-DO: rename method to keyrange()? 
    def index(self, **params):
        """
        Retrieves a range of keys from the inverted file.
 
        Parameters:
            start    (Optional) Defaults to first key. 
            to       (Optional) Defaults to last key.
            count    (Optional) Defaults to 'no limit'.
        """
        params = rename_key('start', 'from', params)
        return self.__doTask('index', params)
 
    def edit(self, **params):
        """
        Attempts to lock a record to allow editing. Returns the record or
        raises an exception.
 
        Parameters:
            mfn       MFN of record.
            lockid    Record lock id.
        """
        resp = self.__doTask('edit', params)
        if get_status(resp) == '0':
            return resp
        else:
            raise LockedRecord, 'edit'
 
    def write(self, content=None, **params):
        """
        Attempts to write a record. Returns the record or raises an exception.
 
        Parameters:
            content    The record's content. Must be a tuple, or list,
                       of 2-tuples (tag, value).
            mfn        The record's MFN, or 'New' to add a new record.
            lockid     Record lock id.
 
        Example:
 
            fields = (
                ('100', 'Some value'),
                ('200', 'Another value')
            )
            db.write(mfn=291, content=fields, lockid='xx')  
        """
        if content:
            content = ''.join([
                "H%s %s %s" % (field[0], str(len(field[1])), field[1])
                for field in content
            ])
        resp = self.__doTask('write', params, content)
        if get_status(resp) == '0':
            return resp
        else:
            raise LockedRecord, 'write'
 
    def delete(self, **params):
        """
        Attempts to (logically) delete a record. Returns the record or raises
        an exception.
 
        Parameters:
            mfn       MFN of record.
            lockid    Record lock id.
        """
        resp = self.__doTask('delete', params)
        if get_status(resp) == '0':
            return resp
        else:
            raise LockedRecord, 'delete'
 
    def control(self, **params):
        """
        Allows to create new databases and to perform several tasks on
        existing databases.
 
        Parameters:
            function    The control function to execute ('unlock', 'invert', 'status', 'create').
            create      If function='create', then create={'master'|'inverted'|'database'}
                        creates the specified type of file(s).
            unlock      If function='unlock', then unlock='control' unlocks
                        only the database's control record.
        """
        return self.__doTask('control', params)
 
    # And these are some convenient shortcuts
 
    def invert(self):
        """
        Generates the inverted file.
        """
        return self.control(function='invert')
    fullinv = invert
 
    def unlock(self):
        """
        Unlocks the master file and all locked records.
 
        Parameters:
            unlock    (Optional) If unlock='control', only the database's control
                      record is unlocked; otherwise, also all locked records are
                      unlocked. 
        """
        return self.control(function='unlock')
 
    def get_status(self):
        """
        Returns information about the current status of database files. 
        """
        return self.control(function='status')
 
 
    # This method was not available in wxis-modules, but is useful for cleaning
    # user-supplied queries.
    def extract(self, **params):
        """
        Returns the keys extracted from the passed data, using wxis's builtin
        mechanism, and optionally specifying custom stw, actab and uctab
        parameters. The method is in fact not associated with a specific
        IsisDb instance, though it could be useful to use the same stw, actab
        & uctab parameters used by the present IsisDb instance. 
 
        Parameters:
            data    The string from which to extract the keys. 
            tech    FST technique (4 to extract words).
        """
        return self.__doTask('extract', params)
 
 
 
# Exceptions
 
class IsisError(Exception):
    # Base class
    pass
 
#class ConnectionError(IsisError):
#    # For errors connecting with the server
#    def __str__(self):
#        return "Error while connecting to the database server"
 
class WxisHardError(IsisError):
    # For errors thrown by wxis (execution, fatal, file)
    def __init__(self, error):
        suggestion = ''
        if '|recread/xropn/w|' in error:
            suggestion = 'In other words, WXIS could not write to the disk. Check file and/or directory permissions for the web server user.'
        elif '|dbxopen:' in error:
            suggestion = 'In other words, WXIS could not open the database. Check that the files do exist and have read permissions for the web server user.'
        elif '|unavoidable|recisis0/xrf|' in error:
            suggestion = 'In other words, WXIS found problems trying to write something. Check database path and permissions for the web server user.'
        self.msg = "\n\n    %s\n\n%s" % (error, suggestion)
    def __str__(self):
        return self.msg
 
class WxisSoftError(IsisError):
    # For errors thrown by a script (missing parameter)
    def __init__(self, error):
        self.msg = error
    def __str__(self):
        return self.msg
 
class BadResponseError(IsisError):
    # For ill formed responses (with no wxis error) preventing the use of eval()
    def __init__(self, resp):
        self.msg = "The database server returned an ill-formed response. Check commas, quotes, braces, and brackets:\n\n%s" % resp 
    def __str__(self):
        return self.msg 
 
class LockedRecord(IsisError):
    # Isis_Status different from 0 when attempting to write a record
    def __init__(self, action):
        self.msg = "Can't %s record -- Record is locked" % action 
    def __str__(self):
        return self.msg
 
class DatabaseDoesNotExist(IsisError):
    def __init__(self, dbname):
        self.msg = "The database %s could not be found" % dbname 
    def __str__(self):
        return self.msg
 
# NOTE: check what other specific error codes may be returned by WXIS, described
# in the documents cited above (Elysio, etc).
 
 
 
#########################################################################
# Tests
#########################################################################
"""
This is a simple test of the code, which also shows how to use the API.
 
TO-DO:
 
  * compare the actual output with the expected output, so that errors may be
    automatically detected.
 
  * create a database from textual data (e.g. the usual CDS as .id or .iso)
    Should we have an extra method, load_iso(), using wxis's <import> tag? Not sure,
    since importing/exporting a database should probably not be done through HTTP...
    But for a purely local test this would be no problem.
 
  * show use of actab, uctab, stw, gizmo?
 
  * besides calling wxis, also show how to manipulate the data in Python, i.e. how
    to replace the formatting language:
      - display a list of records
      - display record details
      - display database status
      - use templates ("$"-based substitutions) to format output:
        http://docs.python.org/lib/node40.html
      - also use the usual "%"-based substitutions 
 
  * special case: MARC records (using pymarc)
"""
 
"""
Original usage examples:
 
1) Browse index keys
 
    >>> db = IsisDb('/home/fer/bases/testdb')
    >>> res = db.index(count=10, start='za')
    >>> [term['Isis_Key'] for term in res['terms']]
    ['ZAANEN', 'ZABCZYK', 'ZABRODSKY', 'ZACKS', 'ZADACH', 'ZADACHA', 'ZADACHAKH', 'ZADACHI', 'ZADATCH', 'ZADEH']
 
2) Search -- TO-DO: simplify using functions
 
    >>> res = db.search(query='marsden')
    >>> import re
    >>> titles = [ unicode(re.sub('\^\w', ' ', field['value'][4:]), 'latin1') for rec in res['records'] for field in rec['fields'] if field['tag'] == '245' ]
    >>> titles.sort()
    >>> print '\n'.join([ '(%s)  %s' % (n, t) for (n, t) in zip(range(1, len(titles)+1), titles) ])
    (1)  A mathematical introduction to fluid mechanics / A. J. Chorin and J. E. Marsden.
    (2)  Algebraic aspects of integrable systems : in memory of Irene Dorfman / A. S. Fokas and I. M. Gelfand, editors.
    (3)  Análisis clásico elemental / Jerrold E. Marsden, Michael J. Hoffman ; versión en español, Oscar Alfredo Palmas Velasco ; colaboración técnica, José Antonio Cuesta Ruiz.
    (4)  Basic complex analysis / Jerrold E. Marsden, Michael J. Hoffman.
    (5)  Calculus / Jerrold Marsden, Alan Weinstein.
    (6)  Cálculo vectorial / Jerrold E. Marsden, Anthony J. Tromba ; traducción: Patricia Cifuentes Muñiz ... [et al.] ; revisión técnica: Eugenio Hernández Rodríguez.
    (7)  Integration algorithms and classical mechanics / Jerrold E. Marsden, George W. Patrick, William F. Shadwick, editors.
    (8)  New directions in applied mathematics : papers presented April 25/26, 1980, on the occasion of the Case centennial celebration / edited by Peter J. Hilton and Gail S. Young ; with contributions by Kenneth Baclawski ... [et al.].
    (9)  Student's guide to Calculus by J. Marsden and A. Weinstein. Volume 2 / Frederick H. Soon.
    (10)  Vector calculus / Jerrold E. Marsden, Anthony J. Tromba.
"""
 
 
def test():
    import os
    from pprint import pprint
 
    def display_status(db):
        resp = db.get_status()
        status = resp['database']['status']
        pprint(status)
 
    def display_records(resp):
        """A simple way to display records."""
        pprint(resp['records'])
 
    def section(msg):
        """Displays a header for each section of the test."""
        line = '-'*40
        print
        print line
        print msg.upper()
        print line 
 
    TEST_DB = 'cds'
    TEST_DIR = 'test'
 
    path = os.path.join(os.getcwd(), TEST_DIR)
    testdb = os.path.join(path, TEST_DB)
 
    # create an IsisDb instance
    db = isis.IsisDb(testdb)
 
    # check db status
    section('check db status')
    display_status(db)
 
    #####################################
    section('list some records')
    #####################################
    resp = db.do_list(start=10, count=2)
    display_records(resp)
 
    # create an FST, or use an existing one
 
    #####################################
    section('generate the inverted file')
    # TO-DO: specify actab, uctab, stw
    #####################################
    resp = db.invert()
    status = resp['database']['status']
    if status == 'inverted':   # why is this check here? should it be catched earlier, and throw an exception?
        print 'Database was inverted.'
    else:
        print 'Some error occurred, database was not inverted.'
 
    section('check db status')
    display_status(db)
 
    #####################################
    section('list some keys')
    #####################################
    resp = db.index(start='W', count=10)
    print [term['Isis_Key'] for term in resp['terms']]
 
    #####################################
    section('do a search')
    #####################################
    resp = db.search(query='water', count=2)
    display_records(resp)
 
    #####################################
    section('lock a record for editing')
    #####################################
    from time import strftime
    some_mfn = 10  # arbitrary
    mylockid = 'test %s' % strftime("%Y%m%d %H%M%S")
    try:
        resp = db.edit(mfn=some_mfn, lockid=mylockid)
        pprint(resp)
    except isis.LockedRecord:
        print "Record %s is locked, can't be edited now." % some_mfn
 
    # TO-DO: attempt to edit, delete or write a locked record
 
    #####################################
    section('create a new record')
    #####################################
    fields = (
        ('100', 'Some value'),
        ('200', 'Another value')
    )
    try:
        resp = db.write(mfn='New', content=fields, lockid=mylockid)
    except:   # what kind of exception??
        print 'Record could not be written'
 
    # display the new record's MFN or error msg
    newmfn = resp['record']['mfn']
    print 'Record was saved. MFN: %s' % newmfn
 
    section('check db status')
    display_status(db)
 
    #####################################
    section('retrieve the new record')
    #####################################
    resp = db.do_list(start=newmfn, count=1)
    #resp = db.search(query='')
    display_records(resp)
 
    #####################################
    section('unlock records')
    #####################################
    resp = db.unlock()
    pprint(resp)
 
    section('check db status')
    display_status(db)
 
    #####################################
    section('delete the new record')
    #####################################
    try:
        resp = db.delete(mfn=newmfn, lockid=mylockid)
    except isis.LockedRecord:
        print "Record %s is locked, can't be deleted now." % some_mfn
    pprint(resp)
 
    section('check db status')
    display_status(db)
 
    # TODO: also show how to clean query using Python only
    #####################################
    section('clean a dirty query')
    #####################################
    query = ' water  plants '
    resp = db.extract(data=query)
    newquery = ' AND '.join(resp['terms'])
    resp = db.search(query=newquery)
    display_records(resp)
 
 
if __name__ == '__main__':
    test()

# -------------------------------------------------------------------
# Configuration file for module wxis.
# -------------------------------------------------------------------
 
 
# Host, port and path to access wxis via HTTP.
# Adjust according to your server.
# Example: if wxis is accessible through the URL
# "http://127.0.0.1:8000/cgi-bin/isis/wxis", then you have
#      HOST = '127.0.0.1'
#      PORT = '8000'
#      PATH = '/cgi-bin/isis/wxis'
HOST = '127.0.0.1'
PORT = '80'
PATH = '/cgi-bin/isis/wxis'
 
 
# Path of the directory where the *.xis files live.
# Use an absolute filesystem path, or (better) a path relative to wxis's
# location.
# For example, if the scripts are in directory 'py-wxis-modules' under the
# directory containing wxis, then you have SCRIPT_DIR = 'py-wxis-modules'
#
# IMPORTANT!! IF YOU CHANGE THIS VALUE, YOU MUST ALSO UPDATE THE *.xis FILES.
# (sorry, it seems to be a wxis limitation that requires hardcoded paths in
# <include> tags)
SCRIPT_DIR = 'py-wxis-modules'