Document Storage Project
This is Part 4: Indexing the storage. Indexing in this context is the process of making the storage searchable – so we can just have a simple text box we type search terms in and get results. We’re not talking about the Apache index we set up in Part 3. There’s all sorts of free projects that do this, but alas most of them just provide you with a library that you can integrate into your own programming project. We don’t want a library, we want a pre-cooked system including database, indexing and an interface we can punch words into and get search results back. The product I’ve found is called Sphider Plus. This is a paid-for fork of a project that’s more-or-less died on the vine. I’m using an older version that I found online– as it’s GPL’d there’s no legal risk to this, but I may well pay for a newer version at some point because there’s some interesting features listed in the newer version and it’s cheap enough. Download and unzip sphider-plus:
cd /home/http/search unzip /path/to/sphider-plus.zip
Now to configure things. First off we need to log into MySQL and create a suitable database. Enter your password when prompted:
mysql -u (username) -p CREATE USER sphider-plus IDENTIFIED BY (put a secret password in here); CREATE DATABASE sphider-plus; GRANT ALL ON sphider-plus.* TO 'sphider-plus'@'localhost'; exit
In the settings/database.php, change the following lines as appropriate:
$database="sphider-plus"; $mysql_user = "sphider-plus"; $mysql_password = "(PASSWORD YOU USED ABOVE)"; $mysql_host = "localhost"; $mysql_table_prefix = "";
We want to be able to index PDFs, so edit the line in settings/conf.php:
//Path to PDF converter $pdftotext_path = '/home/http/search/converter/pdftotext';
Edit /home/http/search/converter/pdftotext as follows:
#!/bin/sh /home/http/search/converter/pdftotext.script $1 -
Set up permissions appropriately:
find -name /home/http/search -type d | xargs chmod 700 find -name /home/http/search -type f | xargs chmod 600 chown -R www-data /home/http/search chmod 700 /home/http/search/pdftotext chmod 700 /home/http/search/pdftotext.script
Point your web browser at: http://(host name).local/search and follow the setup prompts to configure Sphider’s database. Once you’ve done that, log into the admin interface and tell it to start indexing: http://(host name).local Assuming everything works, you should now be able to search for PDFs containing text at: http://(host name).local/search/search.php Look under Statistics/Spidering Logs in Sphider’s administrative interface if you need to troubleshoot any issues. Configure Sphider Plus to your liking. You now have a search interface! But…. it’d be nice if we could search from the file browsing interface. Fortunately, we can. Make the following changes to /home/http/assets/header.html:
<div id="commandbar">
<a href="/" id="home">home</a>
<a href="../" id="parent">up</a>
<a href="#" id="refresh">refresh</a>
<form action="/search/search.php" method="get" id="search">
<input type="text" name="query" id="query" size="40" value="" columns="2" autocomplete="off" delay="500">
<input type="hidden" name="search" value="1">
<input type="Submit" value="Search">
Show
<select name='results'>
<option >5</option>
<option >10</option>
<option selected>20</option>
<option >30</option>
<option >50</option>
</select>
results per page
<form>
</div>
<div id="files">
<h2>
You’ll also want to change the stylesheet, /home/http/assets/style.css. Find the parts that relate to #commandbar and change them as follows:
/* command = commandbar button, is active, is greyed out */
#commandbar a,form {
background-position: left center;
background-repeat: no-repeat;
font-size: 82.5%;
margin-left: 0.4em;
margin-right: 0px;
margin-top: 0px;
margin-bottom: 0px;
padding-left: 22px;
padding-right: 1.0em;
padding-top: 0.4em;
padding-bottom: 0.6em;
line-height: 1.5em;
color: #555555;
border-right: 2px dotted #D5E0E0;
}
/* common commands */
#commandbar #parent {
background-image: url('/assets/icons/parent.gif');
}
#commandbar input,#search,#querySuggestList {
display: inline;
}
#commandbar #refresh {
background-image: url('/assets/icons/refresh.gif');
}
If you point your browser at http://(hostname).local, you should now see your text box you can use for searching: 
Try it out! You should be able to search for text within PDFs with this. However, right now all we have is an index to a single PDF we put up for testing. We need some means of scanning pages and uploading them to this server – the subject of our next post….
