CatalisWiki

Esta página es de solo lectura. Puedes ver la fuente pero no puedes cambiarla. Pregunta a tu administrador si crees que esto es incorrecto.
====== Record Merging Algorithm - for MARC records ======

Tomado de http://www.kcoyle.net/temp/merge.html (2007-08-10), luego movido a http://www.kcoyle.net/merge.html. Contexto: http://demo.openlibrary.org/about/lib

<html>
<table cellpadding="7" cellspacing="0" border="0" width="100%"> 
 

 
<tr> 
<td colspan="3"> 
<table cellpadding="7" cellspacing="5" border="0" width="100%"> 
<tr> 
<td class="contentSmall" valign="top" height="107" colspan="3"> 
 

<p><strong>Pool selection (potential duplicates)</strong></p> 
 
<p>A pool of potential duplicates is obtained from the database by searching on the following data elements from the 
incoming record:</p> 
            <ul>
              <li>LCCN (010 a- normalized)</li>
			  <ul><li>The LCCN has the following format:<br>010 	
##$a###79139101#/AC/MN<br>The index should include only the numeric portion. Skip any leading non-numeric characters, and index to either the first blank or end of subfield.</li></ul>
              <li>ISBN (020 a - normalized)</li>
			  <ul><li>The ISBN in MARC records has the following format:<br>

			  020 	
##$a0394502884 (Random House) :$c$12.50<br>020 	 a 074253779X (pbk. : alk. paper)<br>
        020 	a 9780742537798 (pbk. : alk. paper)<br>
The ISBN is at the beginning of the subfield, but other data can follow. Select the first string in the field, including the ending "X" if it is there. Because of the recent change to a 13-character ISBN, it is probably best to convert all incoming ISBNs to ISBN-13. The alternative is to allow an embedded match between the first 9 digits of the shorter ISBN and the longer one.</li></ul>
              <li>First 25 characters of the normalized title (245 a, b)
                - normalized)</li><ul><li>Normalization generally removes subfield coding, punctuation, and diacritics. Depending on how you have handled the conversion to Unicode, the removal of diacritics may not be possible.</li></ul>
                          </ul> 

<p>We should add OCLC numbers to this search. They are a bit more complex because there is no standard place to store them in MARC records. The OCLC number can be identified by the use of "OCoLC" in conjunction with the number. The OCLC number may contain a prefix of "ocm" or "ocl7". Remove the "ocm" or "ocl7" from the beginning of the number for searching and for matching. They are most commonly found in two places:</p>

<ul><li>In the 001/003 of the MARC record<br>
<pre>
001    62525112 
003    OCoLC </pre>
or in an 035 field<br>
035 	__ |a (OCoLC)ocm51050179<br>In this latter case, the OCLC is identified by the (OCoLC) preceding the number. </li></ul>

<p>When all of these searches have been done, you have the pool of records against which the merge algorithm will act. Depending on how searching is done, you may have the same record in the pool more than once. If possible, eliminate those duplicates (or design the search and pool so that they don't happen.)</p>

 
 
<strong>Level 1 Merge</strong></p> 
 

<p>The merge takes place in two stages. The first level merge allows records to merge on a limited algorithm. This merge is 
only possible when both the incoming record and the pool record contain identifiers (LCCN or ISBN -- or OCLC number).  A threshold weight is 
set for each format in the global parameters table.  If a record receives a weight below the threshold, it is not considered 
a match and the record proceeds to Level 2.  If the record receives a weight equal to or above the threshold, it is 
considered a match.</p> 
 
<p>The threshold for monographs record merging is 875 points (weights are listed in Appendix A).  The Level 1 merge is done 
on:</p> 
 
<ul> 
<li>LCCN/ISBN/OCLC#  [If more than one number is available, the higher points value is used.]</li> 
<li>Publication date (from 008 pos. 07-10)</li> 
<li>First 25 characters of the normalized title (245 a, b combined)</li> 
</ul> 

<p>For efficiency, it is probably best to test each record in the  pool against the level 1 algorithm, stopping when a match has been found. This ignores multiple matches in the pool, but if the database is seeded with a non-duplicative source (eg. LoC records) and each new source is matched against the database, the number of duplicate matches should be low (and may even be suspect).</p>
 
<p><strong>Level 2 Merge</strong></p> 
 

<p>If a merge is not obtained at Level 1, then Level 2 steps are performed.</p> 
 
<p><strong>Full title match  (245 a, b - normalized)</strong> 
<br />The title gets different values depending on how "perfect" the match.</p> 
<ul> 
<li>Exact match (whole string matches)</li> 
<li>Embedded match (one title is embedded in the other, left-anchored)</li> 
<li>Keyword match (a percentage of the keywords that match between the titles, with additional points for having the keywords 
in the same order in the title)</li> 
</ul> 
 
<p><strong>Country of publication</strong> 
<br />This is an exact match between values from MARC 008 pos. 15-17.</p> 
 

<p><strong>Main Entry  (100, 110, 111 - normalized)</strong> 
<br />Comparison of 1XX fields from the records. Since not all records have 1XX fields, there are default values assigned when 
one record has a 1XX and one doesn't, and when both are missing the 1XX.</p> 
<ul> 
<li>Exact match</li> 
<li>Keyword match (a percentage, useful mainly for 110 and 111 fields, which are corporate authors and conference names)</li> 
</ul> 
 
<p><strong>Pagination</strong> 
<br />Pagination is derived from the 300 $a field using the highest number found in that string.  Pagination values are 
only used if they are greater than 10.</p> 
<p>Examples:</p>
<pre>

 300 	 $a viii, 235 p. : = 235
 300 	 $a xxvi, 468 p., [32] p. of plates = 468
 300 	 $a 374 p. : = 374
 300 	 $a 4 v. in one box : = 4
</pre>
<ul> 
<li>Exact match</li> 
<li>Match within + / - 10</li> 
</ul> 
 
<p><strong>Publisher</strong> 
<br />Publisher names in the 260 b field are not highly normalized by library cataloging so this field is used only in rare 
cases where a match has not been attained up to this point.</p> 
<ul> 
<li>Exact match</li> 
<li>Embedded match</li> 

</ul> 
 
<p><strong>Weights</strong> 
 
<br />The table of weights for our program allows us to assign negative and positive weights, with up to 5 different positive 

weights (not counting zero).</p> 

<h2>Record Merge Algorithm for ONIX records</h2>
<p>ONIX records appear to have only a small number of fields filled in, so we can assume that an incoming ONIX record will only use the level 1 match algorithm. ONIX records in the database <em>should not</em> match to each other, since each record represents a single publisher's edition of a way. ONIX records <em>may</em> match to MARC records, and getting this match to work should be our goal.</p>
<h3>Data Elements</h3>

<p><strong>ISBN</strong></p>
<p>ONIX records will probably have both a 10-character and a 13-character ISBN. Here is an example:</p>
<pre>&lt;productidentifier&gt;
      &lt;b221&gt;02&lt;/b221&gt;
      &lt;b244&gt;0002154129&lt;/b244&gt;
    &lt;/productidentifier&gt;

    &lt;productidentifier&gt;
      &lt;b221&gt;03&lt;/b221&gt;
      &lt;b244&gt;9780002154123&lt;/b244&gt;
    &lt;/productidentifier&gt;</pre>
	<p>The field coded &quot;b221&quot; with a value of &quot;02&quot; is the 10-character ISBN. The field coded &quot;b221&quot; with a value of &quot;03&quot; is the 13-character ISBN (which is equivalent to an EAN). Depending on how the software is handling ISBNs, either or both of these needs to be stored and indexed.</p>

	<p><strong>Title</strong></p>
	<p>The title is in the &quot;title&quot; field, labeled &quot;b203&quot;. There are various kinds of titles that can be found in the ONIX standard. If there is more than one title, we should prefer the one with the &quot;b202&quot; code of &quot;01&quot;, which means &quot;Distinctive title&quot;.</p>

	<pre>
	&lt;title&gt;
&lt;b202&gt;01&lt;/b202&gt;
&lt;b203&gt;Wealth Protection Secrets of a Millionaire Real Estate Investor&lt;/b203&gt;
&lt;/title&gt;
	</pre>

	
 <p>One &quot;catch&quot; is that the ONIX titles may begin with initial articles, like &quot;The&quot; and &quot;A&quot;. It is common in the library world to index titles with the initial articles removed, following the indicator value in the title field. We either need to create an index of MARC titles with the initial articles left on, or remove the most common ones from the ONIX titles so we can retrieve them with a string match, or something else that I haven't thought of. 
 </p>
 <p><strong>Dates</strong></p>
 <p>The ONIX publication date is in YYYYMMDD format. The MARC coded date (from the 008) is YYYY. We should create an indexed form of the ONIX publication date with just the YYYY portion. </p>

 <pre>&lt;b003&gt;20060901&lt;/b003&gt;</pre>
 <p><strong>Merging with ONIX</strong></p>
 <p>Using these three data elements should give us enough that 1) incoming MARC records will retrieve ONIX records into their pool and possibly match with them and 2) incoming ONIX records will retrieve MARC records into their pool and possibly match. This theory will need to be tested.</p>
<center><strong>Appendix A: Weights for Monographs</strong><br /> 
(The minimum weight required for merging is +875.)</center> 
<p></p> 
<table width="75%" border="1" align="center"> 
  <tr> 
    <td rowspan="3" width="34%" valign="top">LCCN&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;010 
      a</td> 
    <td width="46%">Match on subfield a</td> 
    <td width="11%">200</td> 
  </tr> 
 
 
  <tr> 
    <td>Field present in both records but no match</td> 
    <td><font color="#CC0000">-320</font></td> 
  </tr> 
  <tr> 
    <td>Either record or both records missing</td> 
    <td>0</td> 
  </tr> 
  <tr> 
    <td rowspan="3" valign="top">ISBN&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;020 a</td> 
    <td>Match on subfield a</td> 
    <td>85</td> 
  </tr> 
 
 
  <tr> 
    <td>Field present in both records but no match</td> 
    <td><font color="#CC0000">-225</font></td> 
  </tr> 
  <tr> 
    <td>Either record or both records missing</td> 
    <td>0</td> 
  </tr> 
  <tr> 
    <td rowspan="4" valign="top">Date&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;008 7-10</td> 
    <td>Exact match</td> 
    <td>200</td> 
  </tr> 
  <tr> 
    <td>+/-2 years</td> 
    <td><font color="#CC0000">-25</font></td> 
  </tr> 
  <tr> 
    <td>No match</td> 
    <td><font color="#CC0000">-250</font></td> 
  </tr> 
  <tr> 
    <td>Value missing</td> 
    <td>0</td> 
  </tr> 
 
  <tr> 
    <td rowspan="2" valign="top">Short-Title&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;245 
      a,b</td> 
    <td>Exact match on first 25 characters</td> 
    <td>450</td> 
  </tr> 
  <tr> 
    <td>Non match on first 25 characters</td> 
    <td>0</td> 
  </tr> 
  <tr> 
    <td rowspan="5" valign="top">Full-Title&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;245 
      a,b</td> 
    <td>Exact match</td> 
    <td>600</td> 
  </tr> 
  <tr> 
    <td>Either title contained within other title</td> 
    <td>350</td> 
  </tr> 
 
  <tr> 
    <td>Either title shorter than 9 characters</td> 
    <td>0</td> 
  </tr> 
  <tr> 
    <td>Non match</td> 
    <td><font color="#CC0000">-600</font></td> 
  </tr> 
  <tr> 
    <td>Matching keywords</td> 
    <td>*</td> 
  </tr> 
  <tr> 
    <td rowspan="3" valign="top"><p>Country of&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;008 
        15-17<br /> 
        Publication</p></td> 
    <td>Exact match</td> 
    <td>40</td> 
  </tr> 
 
 

<td>Either one missing</td> 
    <td>0</td> 
  </tr> 
  <tr> 
    <td>Non match</td> 
    <td><font color="#CC0000">-205</font></td> 
  </tr> 
  <tr> 
    <td rowspan="5" valign="top"><p>Pagination&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;300 
        a</td> 
    <td>Match exactly and &gt; 10</td> 
    <td>100</td> 
  </tr> 
  <tr> 
    <td>Match exactly and &lt; 10</td> 
    <td>50</td> 
  </tr> 
  <tr> 
    <td>Match within 10 and both are &gt; 10</td> 
    <td>50</td> 
  </tr> 
 <tr> 
    <td>Match within 10 and either are &lt; 10</td> 
    <td>20</td> 
  </tr> 
  <tr> 
    <td>Non match (by more than 10)</td> 
    <td><font color="#CC0000">-225</font></td> 
  </tr> 
  <tr> 
    <td rowspan="4" valign="top"><p>Publisher&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;260 
        b</td> 
    <td>Exact match</td> 
    <td>100</td> 
  </tr> 
  <tr> 
    <td>Either missing</td> 
    <td>0</td> 
  </tr> 
  <tr> 
    <td>Occur within the other</td> 
    <td>100</td> 
  </tr> 
 
 <tr> 
    <td>Non match</td> 
    <td><font color="#CC0000">-25</font></td> 
  </tr> 
  <tr> 
    <td rowspan="5" valign="top"><p>Main Entry&nbsp;&nbsp;&nbsp;&nbsp;100 a,b,c,d,k,q<br /> 
        &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 
        110 a,b,c,d,k,n<br /> 
        &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 111 
        a,b,c,d,e,g,k,n,q<br /> 
    </td> 
    <td>Exact match</td> 
    <td>125</td> 
  </tr> 
  <tr> 
    <td>Matching keywords</td> 
    <td>**</td> 
  </tr> 
  <tr> 
    <td>Field missing from one record</td> 
    <td><font color="#CC0000">-25</font></td> 
  </tr> 
  <tr> 
    <td>Fields missing from both records</td> 
    <td>75</td> 
  </tr> 
 
 
 

<tr><td>Non match</td> 
    <td><font color="#CC0000">-200</font></td> 
  </tr> 
</table> 
<p>* Calculate weight based on the percentage of full title keywords common to the incoming record and the database record 
(% in common) x 450.  If keywords are in the same order then add 50.<br /> 
** If half or more of the main entry keywords are in common, calculate weight based on the percentage of keywords in common x 
80.  If keywords are in the same order then add 10.</p> 
 
 
</table> <!-- END NESTED TABLE --> 
<!-- filename: copyrights-help-cdl90 --> 
 

</table> 

</table><!-- END MAIN TABLE --> 
 
</html>


{{tag>marc}}