A search engine generally cannot find information located in a password-protected database, so you have to search them manually. To find out what database to look in, where it's located, or if it even exists, check out the Google Directory for a topical listing of databases on the Web.
Another way to find info not available on search engines is through the "robots.txt" file. The file is used by Web site programmers to tell search engines which directories in a website to ignore. Not all sites use Robots.txt files, but many do. The company will use the file to exclude materials they do not want to show up in search engine listings. From time to time this proves interesting. To find out if a website uses a robots.txt file and get a peek at the contents, type: www.<site name>.com/robots.txt
in your browser's Address line. This displays files and folders that hidden from the search engines. If you see something that looks interesting, type the filename into the browser’s Address line to view the file. If, on the other hand, you get an "Error 404: File Not Found" message, it simply means that site does not have a "robots.txt" file.
Run a search. Save the first few returned pages of hits to your hard drive by choose the File menu and select "Save As" for each page of search results. Go through the list at your convenience.
First, create a new folder on your hard drive. Then, perform your search and right-click on any links that look promising. Choose, "Save Link As" and save the link's destination page to your new folder. Now, you can review the pages offline if you like or use Windows Find (or any search tool) to perform full-text searches on all the pages in the folder, en masse.
In both Netscape and IE you can search the contents of a page (or a complete document) very easily. To do so, hit “ctrl” and “F” and a window will pop up. Type a word or phrase in the window and hit “Find” and the active document in your window will be searched for occurrences of the term! This is great for the situations where you clicked on a link, but when you got to the page you wonder: how did I get here? This doesn’t seem to have anything to do with what I was searching for. When that happens, use this trick to locate your search terms on the page.