FOIA documents : New member with an interest in FOIA material and searches

IsaacKoi

British Barrister
I am a barrister in England with an interest in various issues relating to "UFOs". I've previously helped make freely available online in a searchable format UFO documents from the FBI, official documents from Canada (with the permission of the Canadian government) and Australia (with the permission of the Australian government) and New Zealand, plus helped make freely available in a searchable PDF format various out-of-print UFO publications (such as the newsletters of ufo skeptic Phil Klass, after getting relevant permissions). Here are a few links to samples of my previous items online:
http://www.abovetopsecret.com/forum/thread924183/pg1
http://www.abovetopsecret.com/forum/thread730972/pg1
http://www.abovetopsecret.com/forum/thread736264/pg1

I started reading remote viewing books quite a few years ago because of their discussions of certain individuals that pop up in relation to the history of UFO reports. (There's probably more discussion of some of these individuals in remote viewing books than in books devoted to UFOs...).

Anyway, I thought I'd join here to find out a bit more about the current state of play in terms of producing searchable versions of the FOIA documents in relation to remote viewing.

Has anyone produced a searchable PDF archive yet?

I've written elsewhere about fast and efficient searching of large PDF archives of official documents and other material (in the context of UFO material) and think the same principles could be applied to remote viewing documentation without too much effort:
http://www.abovetopsecret.com/forum/thread762746/pg1

I've tried accessing the remote viewing documentation at the link below, but the index seems to be broken at the moment:
http://www.dojopsi.info/stargateCIAdocs/#about

I've also seen relevant CDs of FOIA releases being sold online at:
http://www.stargate-interactive.com/

I'd be happy to apply some of what I've learnt in relation to UFO documents to helping make the remote viewing documentation accessible more easily and quickly.

All the best,

Isaac
 

IsaacKoi

British Barrister
Okay, I've found the CIA's 15 CDs of FOIA documents on remote viewing on the Blackvault website in their original .tif format.

http://www.theblackvault.com/m/articles/view/The-Stargate-Collection#.U6SjW5RdWSo

Has anyone already converted these files to a searchable PDF archive?
 
Hi Isaac,

Tamra sells the set on her web site which you linked to and has provided a guide. I think Paul Smith also has made a guide for them.

The best method I've found for searching them is dtSearch desktop version. It's not cheap but you can quickly index all the documents and can do sophisticated searches of them. I haven't done any searching in a while with it, though.

Daz Smith has done extensive searching of the SG archives and he uses dtSearch as well and is on top of all the ins and outs of it.

As you saying that TBV has made the 15 CD's available online for free?

Cheers,
Jon
 

IsaacKoi

British Barrister
Thanks for your reply Jon.

The 15 CDs are on the Blackvault site, free of charge, but as the original .tif and .txt files.

So it seems that no-one has made a searchable PDF archive freely available online of the CIA remote viewing archive? If not, I may do it myself this week.

I much prefer PDF archives to having lots of .tif or .gif files.

I find using software which builds an index (such as Copernic or dtsearch) useful for some types of searches but generally I prefer using a free piece of software called PDF-Xchange Editor (the older version of which used to be called PDF-Xchange Viewer).

That free software is at:
http://www.tracker-software.com/product/pdf-xchange-editor

(The site offers "pro" versions, but the free version can be used for as long as you like and has a search feature that searches directories full of PDF files MUCH faster than Adobe Acrobat Pro).

It is more limited than Copernic or dtsearch in some ways (in particular, it can only search PDF files) and is much slower because it does not build an index but instead searching each file each time you run a search - but the search results are presented in a way which allows you to see the context in which the search term appeared and you can open each document quickly from the search results pane - allowing you to run through a lot of search results VERY much quicker than with software like Copernic/dtsearch. Basically, if you are happy to let your computer run a search for a few hours (e.g. while you sleep or watch a movie) then you can run through dozens of results in a few minutes.

I wrote an item comparing different search software a couple of years ago on the UFO UpDates email discussion List. For ease of reference, I'll paste the relevant post below.


From: Isaac Koi <isaackoi.nul>
Date: Thu, 8 Mar 2012 23:07:17 +0000
Archived: Fri, 09 Mar 2012 03:43:52 -0500
Subject: Searching UFO Material On Your Hard-Drive


The email below is being posted as background to a few posts
I'll be making in coming months.

I've recently been testing several pieces of software to make it
easier and faster to find UFO material on my computer. I wanted
to see which piece(s) of software were quickest and/or easiest
to use to search through the UFO material on my hard-drive.

This is a project which has interested me quite a bit in the
last few months. I posted about part of this project a few
months, when it was in its infancy, on this List and on some
other forums - including in the post at the link below at (which
referred to the use of one particular piece of software to
search, and gave links to, collections of MUFON Journals, APRO
Bulletins and NICAP publications):

http://www.abovetopsecret.com/forum/thread762746/pg1

My collection of digitised material has been growing
exponentially in the last few years (to include many books,
journals, magazines, official documents, archives of email
discussion lists, catalogues, indexes and other material),
particularly since I have found a few ways to search this
material more efficiently which has caused me to seek to
increase my collection of digitised material.

Obviously, I'm not about to put all this material online, for
numerous reasons (not the least of which are copyright issues).
However, I'm happy to share some tips and techniques which may
help others to search their own collections more efficiently and
effectively.

I had previously been interested in finding efficient ways of
searching for UFO material online. In particular, I spent a
fairly considerable amount of time seeking to develop various
customised search engines (using, in particular, Google's free
Google Custom Search service) to search some of the better UFO
websites in a single search. I was not happy with the results of
those efforts, particularly because the index used by the Google
Custom Search service is more limited than the index used by the
main Google search service. Because I was not happy with the
results, I don't think I bothered posting here (or elsewhere) a
link to the best of the various customised search engines I
made.

Because of the limitations I found with the Google Custom Search
and because quite a bit of UFO material is not available online,
more recently I've been focusing on searching UFO material on my
hard-drive. As I mentioned here a while ago, I was very pleased
a few months ago to find a piece of software (PDF-Xchange
Viewer) which allowed fast searches of multiple PDF files on my
hard-drive (with an ability to specify which file or folders
were to be searched). More recently, I've been comparing that
piece of software with the Copernic Desktop Search software
(helpfully mentioned to me by Chris Aubeck on the EuroUFO List).

These two pieces of software can be found, respectively, at the
two links below:

(a) PDF-Xchange Viewer:

http://www.tracker-software.com/product/pdf-xchange-viewer

(b) Copernic Desktop Search: http://www.copernic.com/


I've found both of these pieces of software useful. I now use
both, for different types of searches.


I'll compare and contrast them below.


I have also tried DtSearch (which seems to be quite similar to
Copernic Desktop Search - but more expensive, at close to 200
dollars), kindly recommended by Maurizio Verga on the EuroUFO
List. I installed the trial version from the link below but had
a problem. After a while, I received an error message stating
that there was insufficient space on my hard-drive to create an
index since a total of about 120Gb was required. This seems
rather excessive to index a 500Gb hard-drive, being over 20
percent of the total hard-drive space. I don't know if the error
message was due to a bug or something to do with the fact I
already have another index (generated by Copernic Desktop
Search) on that hard-drive:
http://www.dtsearch.com/download.html

So, I'll only give some comments on PDF-Xchange Viewer and
Copernic Desktop Search:



(1) Cost

The basic version of PDF-Xchange Viewer is free (and does
everything I want to use it for, including searching large
collections of files) while Copernic Desktop Search is not free
to use in relation to collections larger than 2GB. The cost of
Copernic is not extremely high (at 49 dollars on its website,
but after downloading the free trial I was soon offered the
software for a "special price" of 39 dollars) - but after being
used to free searches online I'm sure this cost may deter some
people.


(2) Types of files searched:

Copernic Desktop Search is not limited to searching PDF files
(and searches, for example, Microsoft Word/Excel files, on my
hard-drive) while PDF-Xchange Viewer (as its name may imply) is
limited in this way.

Since (for reasons outline below) I generally prefer using PDF-
Xchange Viewer, I've recently had a fairly strong incentive to
convert as much digitised material as possible from Word
documents etc to PDF format. Some of you will have noticed, for
example, that I've been seeking to convert the archives of
various email discussion lists to PDF format to enable me to use
PDF-Xchange Viewer to search those archives (amongst other
material).

Of course, it goes without saying (which will not stop it saying
it...) but both pieces of software can only search digitised
information. Neither is going to help with the piles of books
and documents which I haven't scanned. Again, this has given me
an incentive recently to think about increasing the amount of
UFO material which is digitised. That's probably a subject best
left to another day...


(3) Initial set-up time:

Copernic Desktop Search can take quite a while to produce an
initial index. I had to leave one of my computers alone for
about 4 days for an index of its 500Gb hard-drive to be
compiled.

PDF-Xchange Viewer does not create any index - it needs to run
through each specified file/folder each time a search is
performed. This meansit is quicker to set up.


(4) Speed of obtaining search results:

Copernicus is MUCH faster at producing a list of search results.
Results are virtually instantaneous.

A search of a sizeable collection using PDF-Xchange Viewer can
take quite a few minutes (or even hours when I specify a search
of my entire collection of UFO material).


(5) Speed of REVIEWING search results:

I have found it MUCH easier and quicker to go through the
results of searches in PDF-Xchange Viewer.

The search results in PDF-XChange Viewer indicate how many times
the relevant keyword or phrase appears in any particular
document (with a helpful snippet of surrounding words, which
often allows you to eliminate many of the results) and allows
you to click on each one in turn very quickly, with the relevant
page being displayed almost instantly.

Trying to review the results of a search on Copernic Desktop
Search is, relatively speaking, a pain in the backside. There is
a preview window which displays the first relevant occurrence of
a keyword/phrase within a document when you highlight that
document's filename, but I've found that preview window to be
relatively slow and the formatting of text in that preview
window is often almost unreadable.


CONCLUSION:

I much prefer using PDF-XChange Viewer to Copernic Desktop
Search. Generally, I'd rather wait a few minutes (or even hours)
for PDF-Viewer to produce its search results and then zip
through those results very quickly and easily. I can start a
search on PDF-XChangeViewer and carry on with other tasks on my
computer (or simply start a search before going to bed or before
going out for a meal) and review search results when they are
ready. There isn't usually any massive urgency about getting
results of a search regarding UFO material, so I tend to use
PDF-XChange Viewer because reviewing the results of a search
takes up less of my (limited) spare time. To some extent, the
most appropriate piece of software depends on the type of search
- if there are likely to be a lot of results (e.g. for
"astronomer" or "meteorologist") then I'd focus on the
ease/speed of reviewing results but if I'm not sure there will
be many (or any) results then I would do a quick search using
Copernic Desktop Search.

I hope these comments are useful to some of you.

All the best,

Isaac
 

IsaacKoi

British Barrister
I'm currently looking at the comma deliminated .dat files on the CIA's CDs and the associated guide.

It is quite easy to convert the .dat to Excel files and I should be able to add direct hyperlinks to PDF versions of the .tif files once I finish creating the PDF files.

Doing a few searches on the relevant CIA Files references, , I see that there is already an Excel spreadsheet version of the .dat files at the website associated with this forum:
http://www.dojopsi.info/stargateCIAdocs/Remote-Viewing-STAR-GATE-CIA-files-index.xls

Has someone written an explanation of the various columns in the .dat files? Most of them are pretty self-explanatory and some of the others will probably become clearer when I'm going through the documents. At first glance, I'm currently not sure about:
(1) The "Original Classification" field, where the options appear to be "K", "S", "C" and "U". Presumably these are classification levels, such as Unclassified, Confidential and Secret - which should be clearer when I look at examples of each "original classification" and see the stamps on them.
(2) The "Media type" field, which generally (always?) seems to be "PP" in the sample I've looked at.
(3) The "Release decision" field, where the options seems to be "RIPLIM" and "RIFPUB". "RIP" is presumably something like "Released In Part" and "RIF" is presumably "Released In Full". The "Exemption Code" field is presumably for the FOI Act code in relation to any redactions in "Released In Part" documents.

I'll take Document 1 in Part 1 of CD1 as an example:


CD1, Part 1, Record 1
Box ID : SMT001242790
NARA Record Group : 263
RCS Number : 60-84
RCS Item : 7B
Job Number : 96-00787R
Box Number : 1
Folder ID : RDP96-00787R00010001
Folder Title L PROGRESS REPORT NO. 1-PERCEPTUAL AUGMENTATION TECHNIQUES- STANFORD RESEARCH INSTITUTE
Folder Sequence Number : 1
Document ESDN : CIA-RDP96-00787R000100010001-1
Document Sequence Number : 1
Document Title : PROGRESS REPORT NO. 1 PERCEPTUAL AUGMENTATION TECHNIQUES 12 MARCH 1974
Original Classification : K
Document Creation Date : 03/12/1974
Document Page Count : 49
Copyright(Logical Y/N) : Y
Document Type : REPORT
Media Type : PP
Parent ESDN (blank if no parent):
Release Decision : RIPLIM
Release certification Date : 03/23/2000
Exemption Code : SG1D,SG1I
Date/Event (for re-review) : On File
Document Path on Media : Part0001
 

IsaacKoi

British Barrister
I've now converted the contents of the first CD into searchable PDF files and am working through the other CDs with view to making the searchable archive freely available online.

Just to illustrate the search software I mentioned above (PDF-Xchange Editor) here is a screenshot of a search of the documents on CD1 for "ussr":

This is what the search box and search results pane look like (with the search "hits" shown in context):


If you click on any of the results, the document opens in the main pane on the relevant page with the relevant search term highlighted:
 
Isaac,

Thanks for the very impressive outlining and illustration of the options available.

I see now that, at Daz's suggestion, I did convert a large number of SG files to pdf and set up the files so that I have a pdf version adjacent to the txt version (in the folders). This makes it easy and quick to search for something with dtSearch and copy it out. So far dtSearch meets my needs quite well - but I have not used it extensively.

There have been at least two releases of different sets of SG documents, I believe. There are further unreleased SG documents. Whether they will be released, I don't know.

As I say, Daz has done much more with this than I have - hopefully he will comment here. He is also very knowledgeable about UFO files. (Unfortunately I'm short of time to explore this further now.)

Jon

P.S. One question though - what is the best way to extract text from a pdf file and paste it into PowerPoint or Word? I haven't found the conversions of pdf to Word that good - but have just started looking into this.
 

IsaacKoi

British Barrister
Jon K said:
As I say, Daz has done much more with this than I have - hopefully he will comment here. He is also very knowledgeable about UFO files.
Thanks Jon. I hope Daz will chip in.

P.S. One question though - what is the best way to extract text from a pdf file and paste it into PowerPoint or Word? I haven't found the conversions of pdf to Word that good - but have just started looking into this.
Once the PDF document has been run through some OCR software (whether using the built-in function in Adobe Acrobat or using free-standing OCR software) then the text of the document can simply be copy and pasted if you are just copying a paragraph/page or you can use built-in commands in Adobe Acrobat to export the document to Word format. The two main problems are:

(1) Often government FOIA documents are released in a low resolution or the scans are of poor quality copies of documents. The documents on the Stargate CDs are actually considerably better quality than most UFO documents released by the US Air Force, FBI, NSA, CIA and others - so the OCR results I've obtained today are fairly good.

(2) Some pieces of OCR software are better than others. Recent editions of Adobe Acrobat do a pretty good job. Running British Ministry of Defence documents through Adobe Acrobat's OCR function dramatically improved search results compared to the original OCR software used by the Ministry. So, the searchable PDFs I'm producing using Adobe Acrobat have fairly good OCR results - limited by the quality of some of the documents.
 

IsaacKoi

British Barrister
Mmm. I have nearly a full set of the material but for some reason 13 of the small ".dat" files are missing.

Out of about 4GB of data, I'm only missing a few KBs...

The .dat files from the CDs would be helpful in creating a hyperlinked index for the seachable PDF archive I hope to post shortly.

I've seen the index at the link below (and some others) but I'd like the underlying .dat files for CDs 3 onwards to add to the .dat files I've got for CD1 and CD2:
http://www.dojopsi.info/stargateCIAdocs/Remote-Viewing-STAR-GATE-CIA-files-index.xls
 

IsaacKoi

British Barrister
IsaacKoi said:
I've seen the index at the link below (and some others) but I'd like the underlying .dat files for CDs 3 onwards to add to the .dat files I've got for CD1 and CD2:
http://www.dojopsi.info/stargateCIAdocs/Remote-Viewing-STAR-GATE-CIA-files-index.xls
Just make myself clearer, the spreadsheet at the above link (which I presume was prepared by Palyne "PJ" Gaenir) has a column entitled "CIA Disk Volume Label" which lists .dat files ODC_000000173, ODC_000000174, ODC_000000175 etc.

I have the first two .dat files listed in that column (i.e. ODC_000000173 and ODC_000000174) but not the other ones.

Could someone point me in the right direction for those further .dat files or simply (because they should be relatively small) email them to me?

My email address is my username at gmail.com.

I'll also try making contact with PJ via Facebook. I've sent her a friend request on there.
 

IsaacKoi

British Barrister
PJ has now kindly provided the few missing .dat files.

I can now make the full set of 90,000+ pages of searchable PDF files available online for free in the next few days (with a covering post which will embed images of sample pages relating to the Cash-Landrum incident, the Rendlesham Forest incident and other UFO/ET material).

My focus will be on the "enigma files" element of the remote viewing files (and will refer to some earlier discussions of relevant material in books by Paul H. Smith, Jim Schnabel, Jim Marrs, Jon Ronson and others).

For now, as a test of the hyperlinked index for that material, I have temporarily uploaded a searchable PDF version of two files released by the CIA in relation to remote viewing (Project Stargate etc).

This test just includes two of the files (Disc06 and Disc07 of 15 discs) with a hyperlinked index.

If you open the index and search on anything on Disc06 or Disc07 (e.g. "Landrum", "Higdon" or (!) "GALACTIC FEDERATION HEADQUARTERS") then you will find some of the UFO related material from those discs and should be able to click on the file name within the index to open the relevant document.

I'd be interested in hearing whether this works for those that download the current relatively small (600Mb) test archive from this link:
http://we.tl/G5W30Ucmxu
 

IsaacKoi

British Barrister
mscir said:
Are you still making them available? I'd like to d/l them if you are.
After a bit of a delay (due to events in real life and also getting distracted by another ufological project...), I've now made time to post the searchable PDF archive for anyone to download (free of charge).

I've also posted some sample pages and related information. Details and links are included at:
http://www.abovetopsecret.com/forum/thread1074447/pg1

Thanks again to PJ for her kind assistance.

If anyone has any problems accessing, or downloading, the collection then just let me know here or by email (isaackoi@gmail.com)
 
Top