Latest Update: Added link to Doug’s ‘Digital Packrat’ article.

Introduction

The Situation

Some of my favourite non-fiction writers have an amazing ability to spout any number of fascinating and amazing facts about any subject that comes up. Bill Bryson, for example, or Chuck Palahniuk. At first, I just assumed they had utterly incredible memories, and just knew all this stuff. After a while, though, it occurred to me that they were actively finding this stuff, and storing away whatever they came across from day to day.

Researching stuff as and when needed is pretty easy now. Google will find facts about pretty much anything. There’s some skill to using it well, but it’s not difficult. Storing stuff away day to day, though, is still surprisingly challenging. If you want to be a good factual writer, it’s an important thing to work out.

Purpose

So why am I writing this? I want ideas. I’ll suggest a few of my own thoughts, but I’m interested in how other people have approached this. If nothing else, it’ll make a good article for our writing section. Oh, and getting things written down into an article can be a great way to help clarify your own thoughts.

Why It’s Difficult

It’s not something that sounds difficult to do, but I think there’s a few problems here.

Quick Storage

If it takes long for you to note something down, you won’t bother. If it takes too long to file that note away, you’ll never get around to it, and you’ll be left with just a stack of unsorted notes.

Quick Retrieval

When you come to write an article about eggs, you need to be able to find all the snips of info you’ve collected about eggs. And the quicker you can do that, the quicker you can write that article. If it takes too long, or too much effort, you’ll probably give up, and just use what you remember. The article won’t be as good, and your book (”I Really Like Eggs”) won’t sell very well.

Long Term Storage

You’re not storing stuff away to access next week. You might not need some of this stuff for years. That gives you two problems.

Safety

You need the notes to stay safe for some time. If you lose everything when your hard drive dies, or the paper notes go mouldy in a damp drawer, you’ve lost some important value.

Long-term Accessibility

Shouldn’t be a problem for paper, but if you store everything in computer files, you need to be able to access them when you need them. Will the file format you’re using be readable by the computer you’ll be using in ten or twenty years?

Solutions Writers Have Used

I don’t know how most writers approach this, though I’m interested to hear. A couple I do know about…

Chris Bidmead

I’m fairly sure it was Chris Bidmead who wrote about this a long time ago in PCW Magazine (UK computing magazine). He used plain text files, as they’re the only almost universal format. The filenames were just dates with a serial number appended. The first line of each file was in a specific format, with the source, title, and keywords in it. He then had scripts that could search all the files for anything matching specific sources, keywords, etc.

I remember him mentioning at the time that he’d kept the years as one digit, because the system was only designed with a ten-year lifespan in mind. Not sure what he did when he hit that limit. Could be easily enough expanded by using a folder for each decade, I suppose, or just renaming the files.

Chuck Palahniuk

I’ve just read Chuck’s ‘Stranger Than Fiction’, and he mentions his filing system in there – a wall of filing cabinets he stores everything in. It’s in the context of receipts, though, so I don’t know if this he uses something similar for notes too.

Bill Bryson

Bill is probably my favourite fact-based author, and I don’t know how he approaches this problem. He seems too convinced that his computer hates him, though, for it to seem likely that he entrusts it with storing all of his notes. I could be wrong. There’s a few hints in Notes from a Big Country – he mentions scribbled notes on bits of paper, and storing notes in a file labelled ‘Absent Mindedness’, so paper filed by subject matter would seem to be it.

My Thoughts on the Problems

Quick Storage

Most of the stuff I find these days that I might want to store probably comes at me through my computer, so that seems the most sensible place to put stuff for speed’s sake. Printing it out for paper storage would add time and effort. If most of your stuff comes at you printed, you might be pushed in the other direction by the extra time needed to scan or grab digital snaps of the items.

Time needed to store things depends on how it’s stored, and in what formats. Copy and paste should work pretty well to a text editor to store plain text. Lots of apps now can export HTML. Getting stuff into an application like Microsoft OneNote is pretty easy – just copy and paste. Custom-built databases for this sort of purpose might make things even easier.

Quick Retrieval

Full-text search is probably quite important here, in whatever solution you choose. I think it would be important also, though, to be able to do just a keyword search. So you’re not scanning through every article you’ve ever snipped that mentioned eggs, just the ones about eggs.

Long-term Storage

Safety

If you’re using paper, I guess your enemies here are fire and water. For computers, just be very sure of your backups. Enough good backups can protect against pretty much anything, as long as you keep copies off site.

Long-term Accessibility

This is a bit of a thorny point with computers. Your paper shouldn’t go out of date in ten years. Your computer files might. If I store everything in Microsoft OneNote, will the data still be accessible from whatever computer I’m using ten years from now? Will Microsoft still make OneNote? If they don’t, will the copy I have now still work under Windows MegaSplendid 2015, Ultimate Wicked Edition? I have no idea, and I have no way of finding out. Microsoft don’t know either. If OneNote doesn’t sell, they’ll stop making it. If it sells well, they’ll keep updating it, and make sure it upgrades the data with each new version. My data access is being governed by market forces.

PDF files seem pretty safe right now, but will they still be in twenty years? Plain text is pretty much certain to still work, and I doubt HTML will become unreadable any time soon, but there’s limits to what data we can put in to these formats.

If I wasn’t so anal about it, I’d probably just dump everything into OneNote and stop worrying. But I am, so I do worry. Reducing everything to text adds work, though, which damages our Quick Storage, and makes it difficult to store pictures.

Possible Solutions

OneNote

OneNote wouldn’t even be here, except that I use a tablet PC. OneNote goes well with tablets. At the touch of a button, a nice new window will pop up, sitting on top of whatever else you’re doing, for you to drag stuff to, or make a quick scribbled note with the tablet’s pen. That makes a big difference to your Quick Storage. You can search pretty quickly and easily too, though you can’t really limit it to keywords. The clever part is that you can scribble your notes in ink with the pen, and it can still search those notes. It reads your handwriting in the background, so it knows what the text says.

The problem, though, is the question of how long the data will last. If we’re looking for a solution that will last for ten or twenty years (or more), there’s no way of knowing if it will do the job. And if not, getting five years worth of data out of it all at once five years down the line could take some time. If there’s any chance of you switching platforms somewhere down the line – to Mac or Linux, say, then OneNote is unlikely to ever work on them – though if the much-rumoured Mac tablet ever actually happens, Microsoft may be tempted to port OneNote to there.

Plain Text Files Only

The safest option for the computing paranoid. Can be easily moved from system to system – no problem if you move to Mac or Linux, or whatever else should come along. Should certainly be readable in ten or twenty years.

You’re limited in what you can store – no pictures, no ink notes on the tablet, which does have an effect on how quickly you can store stuff.

Limited File Types

You could choose a limited set of file types you allow yourself to store, and store your data there. To have a consistent way of keeping them dated and tagged with keywords, you could just use a defined format for the filenames – say “YYYY-MM-DD_Note Title_keyword keyword keyword_Source of Note.ext” or something similar.

You can make this as flexible as you like, or as reliable as you like, depending on how many file types you allow.

Plain Text

Certainly should be allowed – simple and reliable, and readable on anything.

HTML

The success of the web means that HTML is readable by almost any computer, and should be reliably readable for plenty of years to come.

JPG

Jpeg images should be as safe as HTML.

Windows Journal

Relevant to tablet users – comes with tablet editions of Windows, so is pretty sure to be in the next version of Windows at least, but could be dropped at any time. Can export to HTML or TIFF images, but this would have to be done for each file, one at a time. Unlikely to be readable in anything other than Windows unless it catches on a lot more than it has so far.

Microsoft Word

Word is probably a settled enough application that its files will still be readable for a long time to come. Readable on Mac or Linux using OpenOffice, if platform-independence is important to you.

Purpose-designed Database

Although it’s not something I’ve really looked into, there are databases specifically designed for storing all these snippets of information for writers. The problem with these is that you would have to be pretty sure that the publisher will still be updating it for as long as you need it to work. I don’t know any of them that well. It also doesn’t seem to be such a unique thing to need to do as to require a special program writing.

Conclusions

I’m interested to hear how other people have tackled this same problem, but my own temptation is to stick to a few file formats with carefully defined filenames to make the searching easier. As a tablet user, though, the temptation of ink is strong. I think maybe OneNote and Journal are great for transitory stuff – to-do lists, making quick notes, planning and writing articles (I’m writing this in OneNote now) – but less useful for any sort of long-term storage.

I think I’ll probably go for lots of files, with just certain file types ‘allowed’ (though it’s my rule, so I can change it) and using a filename that includes keywords to make searching quick and easy.

Any ideas?

External Related Articles

• Digital Packrat on DIY Planner.