using iTunes without losing control of your music/files

Are you like me in that you see some advantages to using iTunes for managing your music collection and synchronizing it with your various Apple devices, but are completely underwhelmed by its basic functionality?  If so, this entry may be able to help you.  If not, then… well… not.

I have a huge collection of music on disk (~300GB) that I maintain manually and am very anal about.  I like the way my files are organized, I like being able to reorganize and re-tag them whenever I want with my choice of 3rd party tools (Tag&Rename, The Godfather, Windows), and I have an extremely anal methodology for pre-processing and merging newly acquired files (files not acquired via iTunes)  to the “master” location.

However, despite my hands-on tendencies when it comes to controlling my digital music collection, there have been several times in the past when I’ve wished I had my entire library present in iTunes (all explain why a little better below).

For a while now I’ve wondered if there’s an “easy” way to have the best of both worlds.  To have my music “in” iTunes while not giving up my low-level control of the actual files.  To be able to work with my media through whatever means I desire and not have these things interfere with each other.

Turns out there was a solution, at least for me, although the “easy” part might be subjective (I’m willing to go to great lengths to manage my music the way I want to, you may be less so after reading this diatribe).  Through the following I’ll document my journey to playing nice with iTunes, and hopefully this might help some other anal music collector looking to do something similar.

Let’s start off by documenting my situation:

My intended goal:

Start using iTunes as the default “library manager” for all my music.

Why do I want to do this?:

There are several advantages, as I see it, to having iTunes manage my music.  All of these are based on the fact that I’m using Apple devices (iPod Classic & iPhone 3G) to actually listen to my music (the benefits of iTunes as a library manager kinda drop off if that’s not the case).  Some of the advantages:

• When you manually manage music on your iPod sans iTunes, any metadata generated on the iPod is not tracked back into your master on-disk library.  If I have my library in iTunes, however,  using the “sync” option means metadata generated from listening habits on the device is now “backed up” into my iTunes library file and I won’t lose it should I upgrade to a new device (more on that below).
• I also like that by taking advantage of the “sync” feature, I can “backup” things like smart playlists, which I currently have only on my iPod (since they make no sense in an iTunes that has no library, as mine now stands).
• Again, using the sync feature means that any metadata changes I make to my files on disk are automatically carried over to the files on my iPod.  When you manually manage adding music to your iPod/iPhone, you lose this “connection” to your master on-disk library.
• If I buy a new iPod, I can simply plug it in and re-synchronize my entire library with a click (including that oh-so-important-to-me track-level metadata I mentioned above).  When you’re manually managing music on the device, this is not an option.  So, if your iPod dies or gets lost, you have to rebuild the collection on it by dragging and dropping everything back.  Painful.  This is probably the single most important thing to me, as I’ve gone through it before.

Success criteria:

1. I don’t want the physical structure of my files on disk to be modified in any way.  I don’t want anything renamed, reorganized, re-anything.  I want my file structure on disk to remain completely unchanged through this big iTunes import.
2. I want to retain “manual” control of my files on disk.  This means having  the ability to rename, reorganize, re-tag, move, delete, etc. these files using applications/methods other than iTunes.  I want to be able to do this without messing up my iTunes library (creating “orphan” entries, out-of-sync pointers, etc.).
3. In addition to the manual control I’d like to maintain in #2 above, I’d also like to be able to use iTunes to manage files if and when I feel like it.
4. If, per #2 above, I’ve manually deleted, moved, or changes files using a 3rd party application or Windows (in other words, not using iTunes) I need iTunes to be “smart” enough to sync up with what’s now on-disk.  In other words, I want my iTunes library to always mirror my manually-controlled library on-disk.
5. Since I don’t get my music from the iTunes store, I also need iTunes to be “smart” about watching my master folder on disk for newly-added files.  When it finds something new, I expect it to be added to the library.   Again, in other words, I want my iTunes library to always mirror my manually-controlled library on-disk.

I know, it may seem picky.  But that’s what I want.

Resolution:

So, was I able to meet all these criteria above and successfully “merge” my music-management process with an iTunes-managed library?  I sure was.  For those interested, I’ll walk through the criteria and let you know how I achieved each one to my own satisfaction:

1. iTunes actually plays pretty nicely here, allowing you to uncheck the “Keep iTunes music folder organized” and “Copy files to iTunes music folder when adding to library” options under Preferences|Advanced (see image below).
   For me, I needed to uncheck both of these (the former to satisfy my criteria #1, the latter to satisfy my criteria #2).  Criteria met.
2. Unfortunately, when you use iTunes to manage your music – iTunes assumes it’s the lone player in this game.  You can go modify files on-disk to your heart’s content, but none of these changes will be reflected in iTunes unless you re-add the modified files to your library.  This is the main problem I have with iTunes, it refuses to acknowledge you might do things to your music outside of iTunes… I did, however, manage to meet this criteria, for the how – see “the panacea” statement below.
3. This one is easy if I can meet 2, 4, and 5.  All I need is for iTunes and my manual processes not to step on each other.  This is more a requirement of my manual process than of iTunes.
4. Again unfortunately, Apple expects that if you’re adding or deleting music, you’re doing it via iTunes.  It makes no accommodations for music file management via other means.  For how I solved this, see “the panacea” below.
5. Again again unfortunately, Apple refuses to accept the fact that people might get their music from anywhere other than the iTunes store, and iTunes is incapable of monitoring folders in this way.  Fortunately, there are some 3rd party applications that can do this for you –  See “the panacea” below.

The panacea: How I met criteria 2-5:

So, how’d I do it?  Believe it or not, there are a couple free pieces of software out there on the great wide internet that can accomplish these things for you (and me).  iTunes Library Updater and iTunes Folder Watch.  Both of them do essentially the same thing, but I’ll tell you below why I chose the former of the two  as my solution.

First up, what do these applications do?  Simply, they allow me to force iTunes to monitor a folder or multiple folders on disk for changes, and synchronize my iTunes library to the folder(s) (without modifying the files themselves on disk).  While this is exactly what I want, there is one major difference between the way each operates.

While both applications keep track of newly added or removed files in your watched folder(s), only iTunes Library Updater includes the ability to monitor track-level metadata (ID3 tag) changes as a synchronization criteria.  For instance, if you’ve used a 3rd-party application (Tag&Rename, etc.) to change some tag data or add album artwork to your MP3s on disk, iTLU will notice this, and update the files for you in iTunes.  iTunes Folder Watch does not include this level of functionality.  So, if you change ID3 information outside of iTunes, you have to acutally play or “get info” on the updated files in your iTunes library to get a refreshed version of the tag information.

For this reason, I consider iTunes Library Updater the better solution, and will continue explaining my “panacea” in the context of using it rather than iTunes Folder Watch.

To make this whole thing work, you simply configure iTLU with the folders you want to be automatically monitored and mirrored into your iTunes library (along with several handy advanced criteria regarding the actual sync process) and hit “go.”  And, since iTLU comes with a command-line interface, the whole process can be scripted into a batch file that you can add to your Windows Scheduled Tasks (I won’t write detailed instructions for this, as someone’s already done a great job over here).  It’s in this way that I have my iTunes library automatically synchronized to my master on-disk library every night while I sleep.

Before you move along here, I would be remiss if I didn’t note the fact that the iTunes Library Updater application is no longer being actively developed.  In fact, it hasn’t been updated since something like iTunes version 6.  So, the fact that it continues to work with the current version of iTunes (versions 8.x as of writing) is pretty much gravy.  Bottom line, future changes to iTunes could completely break iTunes Library Updater at any time. This is a risk I am taking.

My thoughts are that if and when this day comes, I will simply have to switch to iTunes Folder Watch, wait for Apple to implement in iTunes, break down and code something myself, or give up on iTunes for good.

OK, but what’s the process?:

Wow, you actually read this far?  I’m shocked you’re still with me.  If you’re way down here, you probably have a good idea how I did my whole migration, but just to make it simple I’ll sum it up in some easy-to-follow steps for you:

1. Download and install iTunes Library Updater.
2. Configure iTLU to point to the music folders you desire it to mirror into your iTunes library, and check whichever options you want.
3. Save your configuration as an iTLU “profile” to ease repeated future use.
4. Hit “Start” in iTLU and crack a beer as the application imports everything into your iTunes library.
5. Make a batch file that invokes iTLU using the profile you saved in step 3 above (using the console version’s /p switch), and add this batch file to your Windows Scheduled Tasks as often as you’d like (I do it every single night).
6. Continue enjoying your Godlike control over your digital music files while simultaneously taking advantage of the benefits that having your collection in an iTunes library brings.

Conclusion:

And, as for getting your music into iTunes, that’s it.  If you’re satisfied then I think you can stop reading now.  But, because I tend to write a lot… I actually have a few more points to make that, if you’ve come this far, you may be interested in reading.

If you’re done with me here, I appreciate you hanging in there.  If you’ve got comments or suggestions (particularly if you’ve got some super-simple “better way” than what I’ve outlined here), please don’t hesitate to drop a comment below and let me know.  Hope it helped, take care.

Some further notes (for the really anal):

First, a big postscript to all you other “manually manage” people out there.  I know you, you’re just like me.  As you read this, there are exactly zero files in your iTunes library.  You drag-and-drop everything you need right onto your iPod and only use iTunes as the facilitator for that.  But, if you’re an advanced listener like me, you might also have some on-iPod smart playlists defined that do super-fancy stuff with the tunes you’ve manually added to your beloved device.   And, one disadvantage of the “convert to iTunes” method I’ve outlined above is that you won’t be able to import any metadata that’s been generated on your device (through your regular listening) into your new iTunes-based library.

You may be thinking it would be nice to “retrieve” this data before bringing your on-disk collection into iTunes, so that you can continue to build data around your listening habits and drive your smart playlists.  And, you’re right, it totally would be nice to do this.  I, however, made a big concession on this point when doing my huge “gonna start using iTunes” decision.

See, I too, have always said that if I were to make the conversion to managing my entire library via iTunes I’d like to be able to preserve all the metadata I’ve accumulated listening to my manually-added tunes on the iPod (meaning I’d like to be able to keep things like “play count,” “skip count,” “last played,” etc.).   And, you actually can do this.  Yes – you totally can.  See, you can use a piece of software called CopyTrans to first grab all the music off your iPod and into iTunes.  As it does this, it will copy over your precious metadata too (it works, I tried it).

The only issue with this is the extra overhead it adds to the above process…  See if you’re a “manual manager” like me, once you’ve extracted the files (and their associated metadata) off your iPod and into iTunes (which also means they physically went somewhere on disk), you’re going to have duplicate copies of those files in your master collection.  And, since the metadata you want does not “live” inside the MP3 tag data itself, but rather in the iTunes library XML file – you’ll now need to somehow get rid of those metatdata-less dupes in your mater collection (in favor of the metadata-associated versions now living on-disk from the CopyTrans import).

Your head hurt yet?

To try and simplify the explanation, you now want to keep the iPod-extracted files and remove the duplicate files from your master collection.  This is a pain, especially if you have a 160GB iPod Classic like me.  Oh, it’s doable… it really is, and you could keep and carry-over all your on-iPod generated metadata… but it’s just too much work.  So, in my case, I just decided that my big conversion to iTunes would be accompanied by a loss of all my on-iPod metadata.  That simple.  Kinda crappy, but better, in my opinion, than the time-consuming multi-step alternative outlined above.

All part of the paradigm shift, I figure.

Second, I can’t help but assume that, if you’re reading this article and your digital music collection means this much to you that you may also be facing another problem I’ve worked through.  Specifically, now that you’ve got your entire collection into iTunes, how the heck do you manage what actually goes on your iPod and what doesn’t?  It’s the classic “my iPod is too small for my music” problem.  I’ll try and address that in a future entry, because it’s just too long to work through were it tacked on here.

That’s it!  Thanks for reading, take care!

---

Also written on this day...

• a wet-dream supernova - 2010
• change of plans - 2008
• Cheers from Oktoberfest - 2007
• mistaken identity I - 2004
• human bug-light - 2003