I remember reading a designo tract years ago suggesting you group your books by color, by size, or by the publisher's insignia, the latter of which I found most intriguing for some reason. Imagine all the O'Reilly and Penguins and Modern Library books clumped together.
Another way to arrange your personal set of books would be by autobiographical timeline--when did you acquire them? What associations and nostalgia would they bubble up in you? (I think I got that idea from "High Fidelity.")
I have 3 vertical bookshelves in my home office, 2 out in the room, 1 in a closet with the record collection. After a lifetime of grouping books by author or genre, I went a few years ago with a totally randomized approach. I just threw them on the shelves in no order, two-deep. Periodically, when I got too familiar with what was on the top 2 shelves, I'd switch them out with books from the lower shelves. I think I did this because I enjoyed being surprised by finding a book I'd forgotten or enjoying the juxtaposition of 19th-century diarists shelved next to "The Mole People." It broke down the categories in my own head so that I had to keep seeing the books anew.
But it did become too much work to find the book I was looking for and I often found myself tearing the shelves apart when hunting for a specific title. I loved browsing my shelves but hated trying to find something on them.
Inspired by Marc Brodsky, I'm purging my books so that I can only keep what I have shelf space for. (Marc purged his entire collection down to what would fit on a 2-foot shelf, but I'm not that strong.) It's an arbitrary limit, but aren't they all? It's a practical limit anyway.
Lord Peter Wimsey says in one of his stories that one's library is like a carapace, a shell we carry with us that reveals signs of our travels, interests, and philosophies over the years. I'm finding lots of categories of books that I don't need or have time for or have lost interest in, which seems kind of a shame, in a way. As a result, most of my collection is sitting in piles on the floor of my office.
As I re-shelve, the closet bookcase becomes the main Holder of The Books. I'm putting them back in rough genre/subject matter/author clumps: journals/diaries/letters, reference, essays, computer, etc. Art books tend to go on the bottom shelf, which has the most headroom, though all my Delacroix books (his journal and letters and various monographs) sit together in one place, as Hinar described. (Reminds me of how The Book Shop on Franklin Street does it; all of the biographical or other material on a writer is shelved with that writer's novels and stories, so you don't have to go all over the store to find the books dealing with an author.)
One bookshelf is devoted totally to my graphic novel collection, which are arranged by creator (all the Alan Moore stuff in one place, all the R. Crumb in one place). Anthologies are all grouped together. And then within those clumps, pretty much random. I'm not big on alphabetizing by author/title/date/etc. I know geographically about where a book should be, and if it's in that region, I'm happy. The remaining onesie-twosie books are non-clumpable, and therefore randomized. The top two shelves hold unread or unprocessed books/comics/magazines.
The 3rd bookshelf has a shelf dedicated to current schoolwork/papers/registration junk, with other shelves holding most of the fiction and poetry. I tend to group authors together, but not alphabetically. For poetry, I tend to group them on a timeline from ancient sources (Greek translations, through to India, China, Japan) to modern (Wright, Rexroth, Sexton). I never noticed that till I wrote that sentence and I have no idea why I do it.
The top shelf holds the books I'm currently reading (or was reading before school threw itself bodily into my path). When I put a book I'm reading back on the shelf, I place it on the far left. Books I've not read recently migrate to the right, over time. So when I have time to read something, I'll reach for the leftmost book first; I don't have to stop and wonder where that book I was just reading went to. (When I stop reading a book, I either stop at the end of a chapter or stop so that I start reading again on the first full paragraph of the left page.)
It would be a good idea to leave about 10-20% room on a shelf for more books, but that ain't gonna happen.
Aside: My personal book purge makes me wonder -- wouldn't it be interesting to junk a public library's classification system every 75 years or so, and start over again with a new system based on the learnings and experience gained from using the old system(s)?
Other links of interest:
Good Questions: How To Arrange My Bookshelves?
bookshelf on Flickr - Photo Sharing!
Superpatron - Friends of the Library, for the net: Books arranged by colour
Books arranged by colour on Flickr - Photo Sharing!
Huddersfield Public Library Reading Area on Flickr - Photo Sharing!
Huddersfield Colour Coded on Flickr - Photo Sharing!
The library labeled their color-shelved books as the serendipity shelves.