Thursday, September 29, 2011

Mediawiki for personal notes

For years, I've had the problem of how to store my notes but finally found something that is usable.

I've tried a few variations over time, but none suited my use-case sufficiently. Textfiles are too limited and become unmanageable quick. Org-Mode requires Emacs (Emacs and I disagree on how much pain I'm willing to accept in my hands), Zim wasn't available on RHEL and didn't scale well after a while. TomBoy and Gnote require GNOME which doesn't work on the Mac*, aside from me always running into synchronization issues using it on three machines. I toyed with ideas like Evernote but keeping notes in the cloud makes me slightly nervous and doesn't work for confidential stuff anyway. Other wikis I tried over the last few years were dokuwiki, ikiwiki and tiddlywiki.

Many moons ago, Aaron Stafford showed me how he used MediaWiki. At first that didn't seem viable since it didn't cover a few things I wanted: portability, git-based backups that I can take to work and back and use on any machine I care about. Anyway, I've used this for well over a year now and it works for me.

The ingredients:

  • one usb stick

  • truecrupt (realcrypt in Fedora's rpmfusion)

  • mediawiki

(Note: I'm not giving step-by-step instructions because if you're setting this up, you should understand what you're doing instead of just copying commands from a random website)

Preparing the USB stick is easy enough. Format it with FAT32 (if you're not suffering from having a Mac, you can use a better file system). Create a massive file with truecrypt, encrypt it. You can encrypt the entire stick but having the in-file option allows multiple encrypted files on the same stick.

The filesystem for the encrypted file should be FAT32 again. Then install mediawiki in a directory, point apache to it and run through the mediawiki install process. Set the mediawiki up to use an sqlite database. Finally, set up a cron job that essentially runs git add * && git commit -am "Autocommit $date" every hour. Backup is simply done by running git push remote.

I recommend writing a few scripts that automount the stick once you plug it in. Once that's done and on all machines you care about, you just need to plug the stick in, start httpd and off you go. Of course, if you trust your hosting provider enough you could also set it up somewhere on the web and you can skip the USB stick madness.

Now, is this setup perfect? No, by far not. Issues that I see with it:

  • no text-based backend. Now that my X server is decidedly more stable that doesn't matter that much anymore

  • interface decidedly Web 1.0 (TiddlyWiki is much nicer here)

  • no real database merging, essentially requiring a single install instead of several synchronising ones

  • the mediawiki syntax is random at best, and chaotic at worst. As you get used to it becomes less of a problem

  • search hardly works. Not sure if that's a sqlite issue or a broken setup

Lessons learned

Forgetting the USB stick at work means you can't take notes at home for that evening or weekend. I now have a repeating event in my calendar to remind me to take it home.

Databases don't merge and git won't help with a binary file. For a while, I kept two wikis, one for work notes, one for private stuff. But then you have something that overlaps both (e.g. a computer setup that you use at work and at home). Eventually I ended up dumping the smaller database and importing it into the other one before I had too much overlap.

Categories are awesome. They're essentially tags, add [[Category:Foobar]] to any page and the matching Category page lists all pages in that category (plus, that page can also be a page with other info). Categories can be nested.

Everything must be written down. This is something of a golden rule for any note-taking attempt. Unless you take notes, your notes won't be useful.

I still use a normal pencil and paper notebook. Especially when debugging I use pen and paper and then transfer the final result over to the mediawiki.

Install Lazarus. It has saved me a few times.

Redirects help finding pages. Whenever I search for a page with a certain title and it's not there, I add a redirect from that title to the real page. So next time, I'm sorted.

I'm now re-learning how to navigate pages instead of just searching through them. This has one big benefit: on the path to a page you may encounter another page you've forgotten about and that you can now deal with again.

Keep a "Log" (or "Diary") page, with short comments of what you did on each day. I've set this to my home page now and it consists of entries like "Looked at [[Fedora 16 Blocker Bugs]]". It's a quick way to jot down what you're doing, pointing to the pages that contain the actual information.

As said above, I've been using this setup for quite a while now. It's not perfect but software hardly ever is. The final ingredient though I only found last week: the LACIE MosKeyTo, a USB stick that sticks out by only a few mm. Before that I was always worried about breaking off the stick when I carried the laptop around.

So if you're looking for note-taking software, I can't say mediawiki is perfect. But it is the most usable one I've found for me.

* My Mac is essentially a netbook and I don't really install anything on it. It keeps me from working too much at home.


Jonathan said...

You might like the following (currently available from contrib/mw-to-git in git's "next" branch):


install contrib/mw-to-git/git-remote-mediawiki ~/bin/

git clone mediawiki:: -c remote.origin.pages=X_keyboard_extension

Anonymous said...

Inspiring! Currently I encrypt a single text file of personal data and take it with me, have a browser app tab running mGSD (a variant of TiddlyWiki with dependent "getting things done" tasks in contexts), and abuse my User: pages on wikis for public notes; the overlap is frustrating and inconclusive. I'll try your combined approach, despite the challenge of your bare-bones explanation ;-)

What fell short for you in TiddlyWiki? Its single HTML backing store is more compatible with VCS than database files, and I hope someone figures out how to integrate it with Mozilla Sync so that when you access your encrypted browser data in the cloud, your Tiddlywiki file syncs along with bookmarks, history, and passwords. TiddlyWiki shows that a sophisticated app can run entirely in the browser, without needing a separate server. Few realize this is even possible because it has no "social web 3.0" angle to monetize.

Unknown said...

Zim looks awesome. Now if only KDocker would work for it.
There is this attempted org-mode clone for vim called VimOrganizer, but I think it is asleep.

gavenkoa said...

I use org-mode + Mercurial. I keep in sync my notes my merging in Mercurial. So I carry USB stick from home to home...

treegb said...

Wow, you have many similar place with me.
I followed just the same path as you until now.
I'd have spent significant of time to find a knowledge management system/Wordpress ,Emacs Org-mode, TomBoy, Zim, Tidlywiki ...
I don't use Truecrypt but Cryptsetup (Truecrypt have patent issue I saw from other post).

I start to abandon CMS (content manage system) idea and start to write my local html site (static site) because database is not easy to share data compare to html files in filesystem.

After that I start build all my local website manually with stupid way but at least it works
I think connecting html with filesystem will be very useful but both Chrome and Firefox can't do this by default because security concern.
So I write some simple javascript and shell script and modify firefox protocol settings so when I click on hyperlink with specific class, firefox pass the url to shell script, then open the local filesystem directory (with Nautilus) directly for me.
Now I can use browser to navigate my filesyetem!

But i found it's akward to manage this system by hand I'm tired of writing <a href="./foo/bar.html" class="foo">bar</a> all the time. I feel I should not invent the wheels again.

So I came to search something until I found wiki software and mediawiki yesterday.
I tried to study Mediawiki and I found your post. I'm suprised about I just follow the same dark route to find this essential feature!

BTW your teal H1 title WHO-T is as same as my local website title - teal, I love teal.
and I glimpse wacom content in your post, I'm using wacom tablet now! :)))))