Sunday, September 4, 2011

Writing Calibre tags to Openmeta

I've been using the open source book management program Calibre to keep track of my ebook collection. Among other features, it allows downloading of metadata including tags for the books. However, this metadata isn't available outside the Calibre app.

After reading this post on using Python to automatically create tags using Openmeta, I decided to write a script to extract the tags from Calibre and write them to the files using Openmeta.

To run the script, you will need to download the Openmeta binary, and install Calibre's command line tools. This can be accomplished by choosing

Preferences > Miscellaneous > Install Command Line Tools

in Calibre.app or add /Applications/calibre.app/Contents/MacOS to your $PATH variable.

 

The script uses the `which` command to locate the required binaries. It makes a system call to the calibredb program to get the list of books in the current library and extract the tags. Then it calls the openmeta program to write the tags.

import os
import sys
import subprocess
import re


def runshell(command):
    """
    Runs the given shell command and returns an array containing
    the lines from stdout
    """

    p = subprocess.Popen(command, stdout=subprocess.PIPE)
    (stdout, stderr) = p.communicate()
    return stdout.splitlines()


def quotedspaces(array):
    """
    Takes all the array elements that have spaces and returns a new array with
    those elements quoted
    """

    n = []
    for a in array:
        if ' ' in a:
            n.append("'" + a + "'")
        else:
            n.append(a)

    return list(set(n))  # removes duplicates


def checkpath(mypath):
    """
    Check to make sure the path exists
    """

    return os.path.exists(mypath)


def openmeta(path, tags):
    """Given a filename and array of tags calls openmeta to tag
    the file with metadata
    print result
    """

    # get real pathname

    print 'setting tags for ' + path.split('/')[-1]
    ar = [openmetapath, '-s']
    ar.extend(tags)
    ar.extend(['-p', path])

    # call the shell script

    result = runshell(ar)
    return result


calibrepath = runshell(['which', 'calibredb'])[0]
openmetapath = runshell(['which', 'openmeta'])[0]
libraries = ['Personal', 'Technical']


def main():
    '''
    Reads the tags from Calibre database and writes to the files
    using openmeta
    '''

    if not checkpath(calibrepath):
        print "The calibre binary doesn't exist at " + calibrepath
        sys.exit(-1)

    if not checkpath(openmetapath):
        print "The openmeta binary doesn't exist at " + openmetapath
        sys.exit(-1)

    calibrecommand = [
        calibrepath,
        'list',
        '-f',
        'formats',
        '-w',
        '100000',
        ]
    lines = [l.strip() for l in runshell(calibrecommand)
             if re.match('[0-9]', l)]
    for x in lines:
        (num, y) = x.strip().split(None, 1)
        files = y[1:-1].split(',/')
        if files[0] == '':
            continue
        metacmd = [calibrepath, 'show_metadata', num]
        meta = (runshell(metacmd)[4])[22:]
        tags = quotedspaces([z.strip() for z in meta.split(',')])

        for f in files:
            if f[0] != '/':
                f = '/' + f
            openmeta(f, tags)


if __name__ == '__main__':
    main()

Sunday, May 29, 2011

Dot products in C

Calculating the angle between two vectors in C with an interesting gotcha!

In order to find the angle of two vectors in $\mathbb{R}^2$, we compute the dot product. In this example, we have two vectors $\bf{u}$ and $\bf{v}$, where the components are stored in doubles ux, uy, vx, and vy.

To find the angle a between them, we calculate $|\bf{u}|$ and $|\bf{v}|$ and the dot product $\bf{u} \cdot \bf{v}$.

double u, v, ux, uy, vx, vy, a;
double pi = 4*atan(1.0);

ux = rand();
uy = rand();
vx = rand();
vy = rand();

u = sqrt(pow(ux,2)+pow(uy,2));
v = sqrt(pow(vx,2)+pow(vy,2));
udotv = ux*vx+uy*vy;
a = acos(udotv/(u*v))*180/pi;

This produces the angle a in degrees, which I needed for this project. What’s interesting about this is that the results vary slightly if we normalize the vectors before computing the dot product.

So, we get slightly different answers using this code:

double u, v, ux, uy, vx, vy, a;
double pi = 4*atan(1.0);

ux = rand();
uy = rand();
vx = rand();
vy = rand();

u = sqrt(pow(ux,2)+pow(uy,2));
v = sqrt(pow(vx,2)+pow(vy,2));
ux = ux/u;
uy = uy/u;
vx = vx/v;
vy = vy/v;
udotv = ux*vx+uy*vy;
a = acos(udotv)*180/pi;

I believe the error (on the order of 10^–5) is introduced due to the differences in rounding at different stages of computation.

Monday, November 22, 2010

iTerm 2 Bookmarks

I mentioned iTerm in my Essential Software post. Since then, development has ceased on the original iTerm and the project has been forked to create a new version. The new project has been getting a lot of work done and has some really great new features.

One thing that I particularly like is the system for adding bookmarks to connect to particular shells, servers, etc. I have bookmarks set up for ssh and scp access to each of my Macs (both from the local network at home and through MobileMe's Back to my Mac).

I also have bookmarks for some of the command line programs that I use on a regular basis. For example, typing ⌘-Ctrl-O opens a new tab in iTerm with the title Octave and loads octave. This saves several steps over hitting ⌘-T to create a new tab, then typing octave to start the program. In addition, rather than having several tabs with the title Users/~, I can see at a glance which program is running in each tab.

maxima-20101122-111737.jpg

When I first set this up, I had problems with some programs not loading. Thanks to some help on the iterm mailing list, I was able to get this working reliably. The solution was to use the command

/bin/bash -l -c maxima

in the bookmark. The /bin/bash -l command loads the shell as an interactive login shell which loads the .bashrc and .profile files from the home directory. The -c maxima tells bash to execute the maxima command after loading.

Iterm bookmarks

Here are a few other commands that I'm using as shortcuts.

/bin/bash -l -c ipython -pylab # python shell with pylab extensions
sudo /bin/bash -l -c port # macports shell with admin privileges

I hope this quick tutorial helps other people. Having these shortcuts handy makes life a lot easier.

Wednesday, November 17, 2010

Pianobar - One less reason to use Flash

Adobe's Flash Player plug-in can do some pretty neat things, but it's a resource hog and more than a little prone to crashing. Yesterday, I learned about a very cool program that allows you to stream Pandora content without using a web browser or Adobe Air application.

Pianobar is a command line utility that streams Pandora from your Terminal. It is very easy on resources, and is controlled by single-key shortcuts with a very nice help system. Right now, it's using about 2% of my CPU and 4 MB of RAM.

If you're using a Mac, installing Pianobar is very easy. This assumes that you have Macports and Xcode installed. From the Terminal, just enter

sudo port install pianobar

and MacPorts will download and install all the required components. When it's finished type

pianobar

to run the program. You'll log in with your Pandora user name and password. If you don't want to do this every time, you can add the following two lines to the file ~/.config/pianobar/config

user = email@address.com
password = pass

If you have Growl installed with the growlnotify command line component, then you can add the following to get notifications when pianobar changes tracks

event_command = /Users/tony/Desktop/Dropbox/Scripts/pianobar-growl.sh

There's also a GUI client based on pianobar available here. I haven't tried it yet, since the command line version works well for me.

If you're using Windows, it seems that pianobar will compile under cygwin. There's a tutorial here.

Enjoy!

UPDATE: I found an even better client. Pianopub not only streams Pandora, but it offers a graphical interface and responds to the media keys on your keyboard. Totally awesome.

Wednesday, November 10, 2010

Preventing GNU Octave from creating core dump files

I've been using GNU Octave quite a bit this semester. It's a great open-source alternative to MATLAB. Recently, I've noticed the proliferation of files named 'octave-core' in various working folders. After I bit of searching, I learned that these files are created whenever octave is unexpectedly closed and contain the variables that were in memory when octave closed.

Generally, when these files are created, it is because I quit my terminal app (iTerm 2) without realizing that octave was running on one of the tabs. In the documentation for Octave, I found the commands to disable this behavior.

By adding the following lines to a file named .octaverc in my home directory (ie. ~/.octaverc) I was able to automatically disable the core files whenever octave starts.

crash_dumps_octave_core(0)
sigterm_dumps_octave_core(0)
sighup_dumps_octave_core(0)

Problem solved.

Sunday, October 31, 2010

1password support

I mentioned 1password in my previous post on Essential Software. I've been using 1password to keep track of all my log-ins since I switched back to the Mac in 2006. As a result, I have over 800 passwords stored, many of which were created using the built-in password generator. I also sync my password file between all of my Macs using Dropbox.

Last week I had some problems with my Dropbox on my Linux workstation at the SimCenter, and in trying to fix it, I apparently messed up the 1password file. When I got home, all of my passwords were missing—all of them. At that point, I panicked. I made backup copies of the data in my Dropbox and also the Dropbox temp files on both Macs. Then I emailed tech support at Agile Web Solutions, the makers of 1password.

I wasn't sure that much could be done, and I didn't expect much from tech support. Surprisingly, I received an email back within an hour asking me to send a diagnostic file from 1password. I did as requested, then went to bed, not sure if my data could be recovered or not.

When I woke up the next morning, I had an email from tech support detailing how to use 1password to restore to an earlier version of the data file. Apparently 1password makes its own backups periodically. With the help of their tech support, I was able to restore to a file that was saved just before my data was corrupted.

We've become accustomed to low quality tech support from companies. I was amazed to receive quick, helpful support instead of useless, scripted responses. 1password is an awesome program, the customer service is also great. I can't say enough good things about this company.

Sunday, September 26, 2010

Essential Software

Setting up a new Mac

When I started graduate school last month, I decided to buy a new MacBook so that I could have my own system with me anywhere. In the process of setting it up, I took a look at the programs that I already had on my iMac and decided which ones were the most critical to getting my work done. In this list, I've omitted the core set of apps that are included as part of MacOS X. Apps like Mail, Safari, iTunes and iCal are an essential part of my workflow, but I consider those default apps to be a given.

The most important software

System Utilities

These are the core utilities and system enhancers that I use on a daily basis. They are the first things that get installed on a new system.

  • Launchbar I used Quicksilver for years, but with the release of Snow Leopard, the lack of updates caused some significant issues for me. I decided to check out Launchbar and was pleasantly surprised. It's a fast, flexible launcher and much more stable than Quicksilver.

  • Default Folder I've been using Default Folder for almost as long as I've been using Macs. It's a handy utility for jumping quickly to frequently or recently used folders.

  • Dropbox 2 gigs of free storage that syncs seamlessly to your Mac/PC/Linux/iOS device. Faster and easier to use than Apple's iDisk. What more is there to say?

  • Evernote I tend to come across lots of neat little bits of information that I need to file away for later reference. Evernote handles webpages, text and images and syncs between all my Macs, iPhone and iPad for free.

  • TheUnarchiver Apple's built-in zip program isn't bad, but TheUnarchiver is free and can handle pretty much any format however obscure.

Programming and Mathematics

  • TextMate Although it hasn't been updated in quite a while, this is the most amazing text editor I've ever used. It brings the power and flexibility of vim and emacs with an intuitive user interface. The included bundles support most popular languages including C/C++, Perl, Ruby, Python, LaTeX, etc. Not free, but worth every penny.

  • MATLAB As a student in the University of Tennessee system, I get a full copy of MATLAB for free. It's an amazing program with an amazing price tag, and I'm very glad I didn't have to pay for it. For those on a budget, Octave is a free alternative that supports most of the same commands and syntax.

  • MacTeX MacTeX is the standard distribution of LaTeX for the Macintosh. It includes a full TeX distribution as well as some very useful 3rd party tools like BibDesk.

  • MacPorts MacPorts is an easy to use package manager for installing Unix software on the Mac. It handles the process of downloading and installing dependencies

    • gnuplot - open source plotting package that works with octave and other programs

    • octave - open source alternative to MATLAB with an almost identical syntax and programming language

    • aquaterm - GUI plotting package that can also be used with octave. Doesn't require Xwindows.

  • Xcode Apple's free developer tools. It took me a while to get used to Xcode, but for C/C++ it makes a lot of things quite easy. I still prefer TextMate for Perl/Python coding.

  • Skim Open source PDF viewer that integrates really well with TextMate when editing LaTeX documents. Much faster than Adobe's Acrobat.

  • iTerm Open source alternative to the Terminal. Highly configurable.

Internet

  • Safari Extensions With Safari 5, Apple added the ability to load extensions without having to resort to hacks. These are the ones that I use:

    • Add to Google Reader - I've been using Google Reader to handle my RSS news feeds for a while. This extension allows me to click the RSS button in the address bar to add the site to Google Reader.

    • SafariRestore - Automatically remembers your last set of windows and offers to reopen them the next time you open Safari.

    • AdBlock - Reduces the number of annoying ads online.

    • FlickrPlus - Adds a few extra tools to the Flickr site.

    • delicious - One-click to add a page to my delicious account.

  • Adium A great, free, multi-platform instant messaging client. I don't IM a lot, but when I do, I use Adium.

  • 1Password - Amazing password storage that integrates with Safari and other browsers and syncs to other Macs and iOS devices through Dropbox.

General Productivity

  • XMind Mind mapping utility. I've used it a few times to get an overview of what I'm thinking about and find the connections that aren't always obvious.

  • iWork Replaces Microsoft Office for 90% of what I do. Keynote in particular is much nicer to use than PowerPoint.

  • Microsoft Excel This is the only part of Office that I install. Word is the devil, but when it comes to working with large datasets, I haven't found anything that works better than Excel.