1. What are the research issues in the Million Book Project (MBP)?

  • Information storage and management -- The MBP when completed will produce approximately 250 million pages or 500 billion characters of information. The storage requirements for the image files will be approximately 50 petabytes--an order of magnitude larger than any publicly available information base. Creating and managing such a vast information base poses many technological challenges and provides a fertile test bed for innovative research in many areas (described below). The MBP is a multi-agency, multi-national effort that will require the database to be globally distributed. For location independent access, this globally distributed database should appear to be a virtual central database from any place around the world. Mirroring the database in several countries will ensure security and availability. The network speeds at the various nodes would be different. Research in distributed caching and active networks would be needed to ensure that the look and feel of the database is the same from any location.
  • Search engines --The search engines of today work on the principle of keyword matching and perform searches in one language at a time. With a large corpus of multilingual data provided by the MBP, along with multilingual summarization and translation tools, a well-directed research effort would be needed to ensure concept- and content-based retrieval of knowledge from across multilingual data.
  • Image processing --The accuracy of Optical Character Recognition (OCR), even in some of the most developed languages, is hindered by the bad quality of the images. This is particularly true for older books and those that use ancient fonts for which the OCR is not tuned. Even the very best OCR accuracy of the order of 98% may not be acceptable in some cases. In order to obtain an improved accuracy close to 100%, advanced image processing research that will perform recognition beyond the character level will be needed. With the availability of large test data from the MBP and the exponentially increasing computing power of the microprocessors, well-directed image processing research would lead to near perfect optical recognizers.
  • Optical Character Recognition (OCR) in non-Romanic languages --The MBP will have considerable content in many Indian and Chinese languages. The development of OCR in many of the Indian Languages is far more complicated. For example, some of the problems in the development of OCR for Indian languages are:
    • There are1500 spoken Indian languages and 17 scripts.
    • Unlike English, where the number of characters to be recognized is less than 100, Indian scripts have several hundred characters to be recognized.
    • Non-uniformity in the spacing of the characters within a word because of the presence of Consonant Conjuncts (vowel + consonant) makes OCR more difficult. Also, the presence of Consonant Conjuncts results in improper line segmentation. Programs will have to do further processing to segment the lines.
    • Consonants take modified shapes when attached with the vowels. Vowel modifiers can appear to the right, on the top or at the bottom of the base consonant. Such consonant-vowel combinations are called modified characters. In addition, two, three or four characters can combine to generate a new complex shapes called compound characters. These characters are very difficult for a machine to recognize.
    • In scripts like Bangla and Devnagari, all the characters in a word are connected by a unique line called shirorekha (also called head line). In these scripts, character segmentation is especially difficult.
    • In south Indian scripts, vowels occur only at the beginning of a word as against the vowels in Oriya, where they occur anywhere within a word. So, the language morphology for some groups of scripts is different from the others.
    • There is no universally acceptable standard encoding scheme for Indian scripts. This necessitates a scheme where the output labels from the OCR system can be mapped to the labels used by the typesetter through a mapping table. Because of the non-availability of quality-segmented data, the recognition rate of the Indian language OCR cannot be pushed beyond 90% using character level recognition. To obtain a higher recognition rate, word level information in the form of dictionary has to be used. A word corpus is essential, but such a word corpus for most of the Indian scripts is not available today. The MBP will be able to provide this important missing piece. Once such data is available, it will be possible to use advances in image recognition to develop the OCR in Indian and other non-Romanic languages.
  • Copyright laws and digital rights management --In the new digital economy, providing democratic access to information while suitably and reasonably rewarding the innovator is possible. The largest repertoire of free software available on the web in many cases has been the outcome of state supported research. This free availability of software has in fact contributed to more developments and hence an exponential growth of knowledge. Even in literary and scholarly publications, authors have experienced increase sales of their work whenever they are made freely available on the web. This is in tune with observations in the new economy that the companies that make more and more of their software freely available on the web, have their market capitalization enhanced. The MBP, with its proposed plans to make a large knowledge base freely available, will provide useful statistics for testing many economic and sociological models.
  • Language processing --The MBP will produce an extensive and rich test bed for use in further textual language processing research. It is hoped that at least 10,000 books among the million will be available in more than one language, providing a key testing area for problems in example based machine translation. In the last stage of the project, books in multiple languages will be reviewed to ensure that this test bed feature is accomplished.

    • Many believe that knowledge is now doubling at the rate of every two to three years. Machine summarization, intelligent indexing, and information mining are tools that will be needed for individuals to keep up in their discipline work, in their businesses, and in their personal interests. This large digitization project will support research in these areas. This will be of greater significance for the Indian languages where new tools for summarization, grammar and spell checking, thesaurus and translation dictionaries need to be developed ab initio.

    The data provided by the MBP with the right research inputs will facilitate the development of language- and location-independent intelligence amplifiers for furthering information creation.

2. Publishers might not give the MBP blanket permission to digitize and make available all of their out-of-print, in-copyright titles, but might entertain requests for permission to digitize specific titles. Is that possible?

    The MBP approach is to request permission for a range of years, for example, everything published prior to 1990. A publisher could specify the cut-off year or, alternatively, specify the list of titles for which they grant non-exclusive permission to digitize in the MBP.

3. What value-added services will the MBP develop and what formula will be used to calculate publisher royalties? When might participating publishers begin to see income from the project?

    The MBP is not developing a for-profit system. All of the content will be available free-to-read on the Internet. Participating publishers will get copies of the digitized books and metadata, and can themselves provide or enable others to provide value-added services to access the digital books. Permission granted to the MBP is NON-exclusive. Reading the case study of the National Academy Press's experience putting their books online free-to-read could facilitate understanding and appreciation of the benefits of this approach. The case study is available at: http://www.press.umich.edu/jep/04-04/pope.html ( Journal of Electronic Publishing , Vol. 4, Issue 4, May 1999)

4. If a publisher requests removal of a title from the database, what fee would they have to pay for its removal?

    The current cost is $200.

5. What university/scholarly presses are participating in the program?

    The National Academy Press has given us permission to digitize all of their books published prior to 1995. We are currently (July 2002) negotiating with MIT Press.

6. What kind of accuracy will the MBP achieve in scanning?

    Carnegie Mellon has established a workflow (based on pilot ?100 book? and ?1000 book? projects) that includes steps to insure capture of high resolution images and essential metadata, post-processing to correct skewing and crop dark borders surrounding the page images, and OCRing to create searchable ASCII text with 98% accuracy.

7. Once you've scanned a title, how soon will you return TIFFs to the publisher?

    We expect/hope to return TIFF images to the publisher in about three months from the time permission is granted, but the time will depend on how long it takes us to locate copies of the books (after permission to digitize them is granted) and how many books are included in the group.

8. Will the TIFFs meet the Print-On-Demand (POD) standards of Replica and Lightning Source?

    The MBP follows the standards and best practices supported in "A Framework of Guidance for Building Good Digital Collections" Developed by the Institute for Museum and Library Services in 2001 and endorsed by the Digital Library Federation in 2002. See:

    http://www.diglib.org/standards/imlsframe.htm
    http://www.imls.gov/pubs/forumframework.htm

    More specifically, our guidelines for data production (excerpted from the MBP NSF proposal and based on pilot projects) are:

    • Bitonal images with a pixel depth of 1 bit-per-pixel scanned at a resolution of 600 dots per inch (DPI). Images will be stored as "Intel" TIFF (Tagged Image File Format) files with the header content specified. The compression algorithm used is ITU (Formerly CCITT) Group 4.
    • TIFF version 5.0 is acceptable. Subject to testing, version 6.0 (or later) may also be acceptable.
    • The initial-capture system includes dynamic thresholding or a similar feature to capture variability of darkness in the imprint and possibly darker (e.g., foxed) backgrounds from decay. Images should be as readable as the original pages.
    • Typical expected data will be provided for most TIFF tags (normally, the data supplied by software default settings). A specification for the TIFF header will be produced to include scanner technical information, filename, and other data, but to be in no way a burden on the production service.
    • Images will be written in sequential order, with corresponding 8.3 filenames, e.g., 00000001.tif as the first image in volume sequence and 00000341.tif as 341st image in volume sequence.
    • Volumes provided to the MBP will be assigned unique identifiers that conform to 8.3 format. The images will be in directories named with the corresponding identifier (e.g., the volume identified as akf3435.001 will have a directory with the same name, and 00000001.tif through 0000000N.tif files within that directory).
    • Images and directories (as specified above) will be written to gold CD-ROM according to agreed upon specifications and using ISO9660 format.
    • Skew will be within a specified range of degrees allowed.

9. Will the consortium formed to address publisher compensation issues have anything to do with the administration of the MBP? If not, how will the MBP be administered? Who will make decisions for it? Will participating publishers have a role in decision making?

    No, neither the consortium nor participating publishers will have an administrative role in the MBP. Administrative responsibility for the MBP belongs to officials at participating universities. At Carnegie Mellon, the administrative officials are:

    • Dr. Raj Reddy, University Professor, School of Computer Science
    • Dr. Gloriana St. Clair, University Librarian
    • Dr. Mark Kamlet, Provost

10. Who will determine the pricing of value-added components of the MBP?

    The publishers or vendors who develop the value-added components will determine the pricing for the services they provide.

11. What content will be included in the Million Book Project? What scanning is currently underway? What about copyright permissions?

    Our initial thinking was to take a staged approach to collection development on a discipline-by-discipline basis. However, discussion with project partners and potential partners in November 2001 at a collection planning meeting funded by NSF resulted in the decision to focus on providing free-to-read access to multiple collections. Copyrighted works will be digitized upon receipt of permission from the copyright holder to include the works in the Million Book Project.

    Our partners in India and China are currently digitizing local materials. Our Chinese partners are digitizing unusual and unique rare collections in Chinese libraries. Our Indian partners are digitizing government textbooks published in eleven of the eighteen official languages in India.

  • U.S. works that aren't copyrighted

    The following U.S. government documents, many of which are available on microfiche, have been targeted for inclusion:

    • Department of Education - 30 boxes from Carnegie Library of Pittsburgh's collection are currently prepared for shipping to India
    • Administration and Management - 27,400 titles
    • Aeronautics and Aerodynamics - 25,400 titles
    • Agriculture and Food - 21,600 titles
    • Astronomy and Astrophysics - 12,500 titles
    • Atmospheric Sciences - 17,010 titles
    • Behavior and Society - 43,600 titles
    • Biomedical Technology and Human Factors Engineering - 9,300 titles
    • Building Industry Technology - 14,600 titles
    • Business and Economics - 31,700 titles
    • Chemistry - 60,100 titles
    • Civil Engineering - 16,900 titles
    • Combustion, Engines, and Propellants - 16,700 titles
    • Communication - 25,900 titles
    • Computers, Control, and Information Theory - 35,300 titles
    • Electrotechnology - 28,200 titles
    • Environmental Pollution and Control - 89,900 titles
    • Government Inventions for Licensing - 8,200 titles
    • Health Care - 10,700 titles
    • Industrial and Mechanical Engineering - 33,300 titles
    • Library and Information Sciences - 34,800 titles
    • Manufacturing Technology - 31,200 titles
    • Materials Sciences - 22,300 titles
    • Transportation - 22,500 titles
    • Urban and Regional Technology and Development - 23,400 titles
    • Bureau of Energy - 62,500 titles (over the past ten years)
    • NASA - 22,500 titles (over the past ten years)
    • U.S. Serials - to be contributed by the TriCollege Libraries

  • U.S. out of copyright works

    As of August 2002, Carnegie Mellon University Libraries is pulling books published before 1923 and boxing them for shipment to India. We estimate digitizing 45,000 titles in the Carnegie Mellon collection, including biographies and science books. The University of Indiana has identified approximately 2000 pre-1923 literary works to contribute to the project. The University of California at Berkeley has identified another 5000 pre-1923 works to contribute to the project.

    We have identified approximately 2000 titles in Books for College Libraries that are out of copyright. The next step is to locate these books in Carnegie Mellon's library collection or the collections of our project partners and ship them abroad.

  • U.S. copyrighted works

    We've received permission from the National Academy Press to digitize all of their titles published through 1994--approximately 3500 titles. As of August 2002, hundreds of these books have been pulled from Carnegie Mellon?s library collection and boxed for shipment to India in the immediate future. The next step is to locate the remaining titles in partner libraries. Those that are not already available in Chinese libraries will be located in the U.S. and shipped abroad for scanning.

    In June 2002 we began sending letters to U.S. publishers seeking permission to digitize all of their out-of-print or remaindered in-copyright works. Negotiations are underway with Indiana University Press and MIT Press. We are preparing to send letters to publishers seeking permission to digitize all of their copyrighted titles listed in Books in College Libraries, approximately 58,000 titles. When permission is granted, we will locate the books in partner libraries. Books that are not available in Chinese and Indian libraries will be located in U.S. partner libraries and shipped abroad for scanning.

    Carnegie Mellon has committed to digitizing the university's many technical report series--approximately 5500 titles. Permission to digitize the reports is being sought from the copyright holders. Scanning the Robotics Institute's technical reports began in August 2002. Permission is also being sought to scan technical reports published by IBM and Microsoft.
  • Other

    We want to digitize the British Parliamentary Papers and are currently investigating whether these documents are copyrighted. We believe that many of these works are available in libraries in India.

12. Will the Million Book Project preserve the fixed format of the initial publications?

    Yes, the digitized works will preserve the fixed format of the initial publications.

13. What metadata is being captured about the digitized works?

    MARC records and administrative metadata are being captured following existing standards. Dublin Core is being used for materials that have not previously been catalogued or where MARC is inappropriate, for example, for photographs and three-dimensional cultural artifacts.

14. Who are the key U.S. participants in the Million Book Project?

  • Dr. Mark Kamlet, Provost, Carnegie Mellon--Kamlet led the delegation to China in 2002 and will lead the delegation to India in 2003.
  • Dr. Raj Reddy, University Professor, School of Computer Science, Carnegie Mellon
  • Dr. Gloriana St. Clair, University Librarian, Carnegie Mellon University Libraries
  • Dr. Ching-chih Chen, Professor, Simmons University
  • Dr. Michael Shamos, Distinguished Career Professor and Principal System Scientist, School of Computer Science, Carnegie Mellon
  • Dr. Jaime Carbonell, Director of the Language Technologies Institute and Professor, School of Computer Science, Carnegie Mellon
  • Mr. Eric Burns, Director of Software Development for the Universal Library (parent project of the Million Book Project), School of Computer Science, Carnegie Mellon
  • Ms. Gabrielle Michalek, Senior Archivist and Digital Projects Coordinator, Carnegie Mellon University Libraries
  • Ms. Denise Troll Covey, Principal Librarian and Associate University Librarian of Arts, Archives and Technology, Carnegie Mellon University Libraries
  • Ms. Erika Linke, Senior Librarian and Associate University Librarian of Collections and User Services, Carnegie Mellon University Libraries
  • Mr. Brewster Kahle, Internet Archive
  • Phyllis Spies, Andrew Wang, and Lorraine Normore, OCLC

15. Who are the other partners in the Million Book Project?

  • China--Representatives from these institutions met with key U.S. participants in the U.S. in 2001 and in China in 2002:
    • Chinese Ministry of Education
    • Chinese Academy of Science
    • Fudan University
    • Nanjing University
    • Peking University
    • Tsinghua University
    • Zhejiang University
  • India--Representatives from these institutions met with key U.S. participants in the U.S. in 2002 and will meet with them in India in 2003:
    • Indian Institute of Science, Bangalore
    • International Institute of Information Technology
    • Indian Institute of Information Technology
    • Anna University, Chennai
    • Mysore University, Mysore
    • University of Pune, Pune
    • Goa University, Goa
    • Tirumala Tirupati Devasthanams, Tirupathi
    • Shanmugha Arts, Science, Technology & Research Academy, Tanjore
    • Arulmigu Kalasalingam College of Engineering, Srivilliputhur
    • Maharashtra Industrial Development Corporation, Mumbai
  • United States--The following institutions have agreed to participate in the early stages of the Million Book Project. Additional institutions have agreed to join the project when we have gathered data on the time and cost of sending materials abroad, having them safely returned, and delivering electronic copies of the scanned materials to the home institution.
    • Indiana University
    • Pennsylvania State University
    • Stanford University
    • TriColleges (Swarthmore, Haverford, Bryn Mawr)
    • University of California, Berkeley
    • University of Pittsburgh
    • University of Washington

16. How is the Million Book Project supported?

    To date, two grants totalling $3.5 million have been received from the National Science Foundation to purchase equipment.

    The Chinese Ministry of Education, Chinese Academy of the Sciences, Indian Institute of Science, and Carnegie Mellon University Libraries and School of Computer Science are providing personnel and facilities, and participating in collaborative research. Carnegie Mellon University Libraries is training the scanning operators.

    Brewster Kahle is providing disk storage.

    OCLC is providing project partners with metadata at no charge, will support a registry to track progress and avoid duplicate scanning, and might become a sustaining host of the final Million Book Project collection.

    Additional grant proposals are planned to support seeking copyright permissions, further collection development, the management of project logistics, and shipping costs.