Using Multiple Folders for iTunes
Posted 21 December 2007 - 05:24 PM
A couple years ago I migrated my entire music collection to my computer and sold all my CD's. Due to the size (250 gigs of apple lossless files) I put it on an external hard drive (then of course backed that up onto a completely separate external drive) which gave me the freedom to not just use the drive with my desktop system, but to take it on the road and use it with my laptop as well. Combined with airtunes optical output, the lossless files sound at least as good as the CD's ever did, and it doesn't matter where I have my computer or my stereo... the wireless signal means that it works perfectly. I love the whole set-up.
So far so good.
Generally when traveling I take my 60gb ipod, which obviously won't hold the 250 gigs of music, but that's okay, since putting the lossless files on there would just eat up more batttery power anyway. So, I made another folder and used iTunes to copy all the music to it, knocking it down to reasonably sized AAC files - Voila, I've got all my music on my ipod with a bit of room to spare for TV shows and such.
For the last two years, I had no problems with this set up.. I'd import a new CD as both a lossless file to the main folder and as an ACC to that one, and make sure I was using the ACC files when I would sync my ipod. But now, suddenly I am having problems. I did a sync a couple days ago and then switched back to my lossless folder, as I've always done, but for some reason, iTunes is still wanting to use the ACC files instead of the lossless. The lossless files are on the drive, same place they've always been, and the advanced preferences are set up to look for them there, but for some reason it stayed with the ACC files in the library. I then tried unmounting the hard drive with the ACC folder on it, thinking that itunes would have no options then but to look in the proper folder for the lossless files. Instead, now it's showing me the exclamation marks for all of those songs, saying it can't find them. I can remount the other drive, and use the ACC files, or I can easily go through, one by one, and point each song to the proper place on the lossless drive, but obviously with 250 gigs of music, one by one is not an option for a true fix, even if it is only a few keystrokes. I've tried and tried to find a way to point the entire library, or at least album by album, back to the right folder, but iTunes is having none of it. It's one song at a time or nothing...
So, yes, I've finally gotten to my question:
Is there a way to point an entire batch or library of songs to a new folder when iTunes isn't seeing it on its own?
And secondly, is there something I'm doing wrong that leads to this problem? I can't imagine I'm the only one out here who is using this kind of set-up, wanting to keep a lossless folder for my computer, where storage space is plentiful and quality is paramount, but also wanting to keep a folder for my ipod, where a compromise between quality and file size are important. This worked smoothly for so long, I've been wracking my brain for the last two days trying to figure out what is different, but I can't find a reason for this, nor a solution.
Any help from those more wise than me, would be deeply appreciated.
edit - (it shouldn't matter too much for this question, but since it will come up, I am using a Mac Pro desktop system running 10.4.9 - latest updates all around, with the latest version of itunes. The drives are a iomega firewire 800 drive, and a WD firewire 400 MyBook drive as the backup. iTunes won't recognize either lossless folder on either drive without being pointed there manually, song by song. )
Posted 22 December 2007 - 08:08 AM
In general, I would recommend creating a folder structure to organize one’s music collection, but in your case separate libraries would perhaps be the best way to go in the long run. The reason for that is because you have music files that have ID3 tags that are mostly identical excepting format and bit rate. Realistically, iTunes should not confuse the Apple Lossless and AAC versions of your files, but as they are near exact duplicates, it is a possibility. The sudden loss of half od your collection—in terms of linking to the music files—seems to point to database corruption.
The biggest problem is that your collection is perhaps quite sizeable making the task of separating libraries a rather daunting task at this point—it would have been easier to do so as you ripped your CDs, which is one of the reasons I spent a great deal of time planning the organization of my iTunes library (e.g., playlists, folders, tagging standards, etc.) before I ripped the first CD then scanned and added album art as I ripped each disc. As your music is essentially a single library with (at least) two folders, I cannot think of why iTunes would find the contents of one folder but not another. If you simply look at your music library in iTunes, as opposed to the specific folder contents, do you still get the missing link icons?
If you tell iTunes to play music from a particular folder, then it should do so. For instance, if I have my “R&B (1990s)” folder selected and start playing music iTunes should not be playing music from my “Rock (1980s)” folder. Of course, I almost exclusively play music directly from iTunes using Party Shuffle, but the same rules apply. I am not sure of what iTunes behavior would be if a song is missing a link to a file, but I would guess that the track would be skipped. In your case, because you have duplicates in another format, perhaps iTunes is intelligent enough to seek duplicate songs; as a result iTunes may be switching to your AAC folder.
In the worst case, you should not need to re-link your lossless files one-by-one, but there are a few things I would need to know before I can determine a feasible plan of action:
* Do you have “Keep iTunes Music folder organized” selected as a preference?
* Do you have your Apple Lossless and AAC files distinctly separated on your external drive? That is, are the two file types in separate folders on your hard drive or lumped together as just files?
* In iTunes, do you only have the two folders based on file type or does each file type folder also have a folder/playlist structure within them?
Once I have the specifics of how you have your music organized in iTunes and on your hard drive, I may be able to offer suggestions on how to recover your music library.
Posted 22 December 2007 - 11:30 AM
Thanks for the reply. First, the answers to your questions:
Yes, I let iTunes keep the folders organized, which thus far has ensured that the two individual music folders are identical in every way except for file size and types (ACC vs lossless).
Yes, the music folders are completely separate. Currently they are on the same drive, but there have been times in the past when they were not - specifically, the lossless files have always been on the same hard drive. At various times it has been more convenient to keep the ACC folder on an internal drive. The location of this folder has never been an issue, as iTunes is either pointing at one or the other, never both at the same time.
I'm not sure I'm reading your third question correctly, but each file type folder has the same multi-folder organization within it. In other words, currently, on that hard disk, there are two folders, one labeled "Compressed Music" one labeled "Lossless Music" if I open either one of them, I see a list of all of the various bands and artists for which I have music. If I go to the folder for "The Beatles" and open it, I see a list of folders for specific albums. If I go to "Revolver" and open it, then I see the list of specific songs, and this is the only place that the two folders differ. One has the file "01 Taxman.m4a" at the top which is 3 mb in size, the other has "01 Taxman.m4a" at the top which is 16 mb in size. This is the first time I've noticed that the lossless and ACC files use the same file extension, which is strange, but nevertheless, this is how the two distinct folders are set up, and this has worked for two years runnning. This multi folder organization by artist and then album and finally song was the way itunes wanted to set things up, and I saw no reason to stop it or do otherwise.
Switching between the folders was (and should be) as easy as going into advance preferences, changing the "itunes music folder location" to the wanted folder (either Compressed Music or Lossless Music) and letting itunes update and organize itself. With this volume of music, switching between folders takes several minutes, but since I only do it once every month or two, when I need to update my ipod with the new CD's I've bought and burned, that time lag is not really an issue.
"I cannot think of why iTunes would find the contents of one folder but
not another. If you simply look at your music library in iTunes, as
opposed to the specific folder contents, do you still get the missing
- Let me emphasize that the music folders are not affected, only the iTunes library is, and only by not pointing to the right places. All songs are still present when I open itunes, just most of them have the exclamation marks on the left side indicating that the program is not seeing the file to which it should be pointing.
"If you tell iTunes to play music from a particular folder, then it
should do so. For instance, if I have my “R&B (1990s)” folder
selected and start playing music iTunes should not be playing music
from my “Rock (1980s)” folder."
- it sounds as though you feel the issue is with playlists. This is not the case at all. Your wording makes me wonder if you've told itunes to organize your music differently, or if you are talking about playlist folders. the playlists have nothing to do with this issue, and nothing to do with how the music is organized on the hard drives (question 3, above).
"I am not sure of what iTunes behavior would be if a song is missing a
link to a file, but I would guess that the track would be skipped. In
your case, because you have duplicates in another format, perhaps
iTunes is intelligent enough to seek duplicate songs; as a result
iTunes may be switching to your AAC folder."
You are right, that if itunes is missing a file link for a song that it will skip that song when playing, but that's just a side effect. iTunes does not look for the original file and then revert to the ACC folder or go there alternately - it just doesn't look for the file in the first place. The music is not missing... (as I mentioned before) once I individually point iTunes to the file it says it can't find, it's happy again. I'm just trying to find a way not to have to do that 6,000 times, or to have to rebuild the library completely (thereby losing all my playlists, song rankings and other information). And to find out what is causing the issue, so that this doesn't happen again.
Incidentally, I too thought this through quite carefully before burning any of my music. This dual file structure seemed the best answer, and worked perfectly in small scale tests in the days before I started the massive process of burning thousands of CD's to hard drive (and worked flawlessly for two years with the full library as well, of course) and in several weeks of searching before hand, I didn't find anyone with a better solution to the issue. I understand your recommendation against getting rid of the original CD's, but that was a big part of the point. I was beginning a job where I would be traveling (and moving) a great deal, and a couple thousand CD's are no small effort to keep, store, organize and move. Selling them off more than paid for the expense of buying a couple good hard drives to store the music, and with the exception of two CD's which I discovered later were slightly corrupted in the burning process, and which I didn't listen to before selling them off (which I did with all my favorite music), there have been no losses or downsides to to this migration at all. Just the opposite... I enjoy my music collection more than I ever did when I had to look through shelves and shelves of CD's to find the one I was looking for.
In addition, with one hard drive sitting on the desk holding my music as well as my other work, if I am home when there is a fire or other incident, unplugging the drive and carrying it out the door with me will save not only my work, but my entire music collection - something that would have been impossible with a couple thousand CDs. I'm currently investigating off site backup as well, and at the very least, am considering buying a cheap hard drive to copy my work and music files and keep at my parent's home, as an easy off-site long term insurance (especially since my neighbors in the next apartment started a fire a week ago!). While in the ideal world, I'd have all the CD's sitting around, the fact is, my music collection is at least as safe, if not safer, like this than it was when I owned the CDs, and I don't have to break my back everytime I move.
Hopefully that didn't sound defensive... I realize that not everyone wants to sell off a massive music collection and just keep it virtually, but my feeling is that just like LP's (aside from a few diehard collectors) most people will head that direction in a few years. For my lifestyle, carrying around a couple hard drives is far easier than a dozen or more milk crates weighing 30 lbs each.
Thanks again for your reply,
Posted 22 December 2007 - 06:27 PM
Where your problem may lay is that while you have two distinct libraries, you are forcing iTunes to use the same library database file fro each of them. Why it took this long for problems to surface is anyone’s guess. The way iTunes is set up, setting a library location through the preferences actually does not change the location of your iTunes library, but instead tells iTunes where your music files are located. In the preference settings you are dealing with the permanent library and that is always located in ~/Music/iTunes. Even if you delete the iTunes Library and iTunes Music Library.xml files in that location and select a location in preferences, new database files will be created in ~/Music/iTunes the next time iTunes is launched regardless of the location of the actual music files.
For switching to an alternate library, as you do when syncing your iPod, you are supposed to select a library by re-launching iTunes and invoking the Choose Library dialog. When you follow that procedure, you actually select an alternate iTunes Library file in a location other than ~/Music/iTunes and the music files associated with that library. If I recall correctly, the ability to (easily) choose alternate libraries was added in iTunes 6.
My guess is that what you are experiencing is the result of forcing your permanent database file to deal with two different sets of audio files. iTunes Library is updated as you use iTunes so when you are playing music from your computer using the Apple Lossless files, the database is no longer a perfect match to the AAC files you offset despite the fact that it is the same songs. At this point you have either corrupted the database file to the point where it no longer sees the lossless files or the differences between what should be iTunes Library for the AAC files and iTunes Library for your Apple Lossless files has resulted in confusion.
Given the way in which you have designed your music libraries what you want to do is have your lossless files recognized as the media for the permanent music library set up in preferences and your AAC library as an alternate library that you choose when you wish to sync your iPod. That is, the drive on which you have the lossless files would always need to be attached and mounted on your Mac, or least when iTunes is running and you would set the preferences to point to the Lossless Music folder because it is your permanent music library. The iTunes support files in ~/Music/iTunes would be linked exclusively to the lossless music files.
For your AAC files you would need to organize the folder the same as the system created iTunes folder at the root level. So Compressed Music would contain the folders Album Artwork and iTunes Music, or whatever you choose to name the latter, and the support files iTunes Library and iTunes Music Library.xml all associated with the AAC files. The iTunes Music (or whatever you name it) folder within Compressed Music would contain the artist > album > song tree of your AAC files.
When you want to switch libraries, you re-launch iTunes and immediately hold down the Option key after launching until the Choose Library dialog appears. When you want to sync to your iPod you would choose ExternalVolumeName/Compressed Music/iTunes Library, but when you want to switch back to your permanent library you would choose ~/Music/iTunes/iTunes Library. In that way you keep your libraries linked to their distinct database files and eliminate “crosstalk” corruption issues, which is what I believe you are now experiencing.
bq. “it sounds as though you feel the issue is with playlists. This is not the case at all.”
Yes, I was not sure if you were referring to file folders or playlist folders based on your original post. Typically when people discuss folders in reference to iTunes, they are not referring to the folder structure on the hard drive; most very likely do not pay attention to how their music files are organized outside of the iTunes user interface. I was definitely talking about playlist folders when I referred to the R&B (1990s) and Rock (1980s) folders in my iTunes library.
bq. “Hopefully that didn't sound defensive…”
Not at all. Explaining why you pitched you CDs and set up your library makes it easier for those attempting to assist you see your perspective on the matter. I have a large collection of CDs and vinyl that I have no plans to ever sell off, but I do not move about often either.
In the end, I hope that you can find a way to recover your bases without having to go song by song. With my collection already sitting at 12,109 songs—and I have not even started the process of digitizing my vinyl collection yet—I feel your pain. Given that your current iTunes Library seems to think that it is linked to your AAC files, I would suggest adding the extra level to your Compressed Music folder as described above then going into the preferences and linking the library to the second level folder to update the iTunes Library file. Once you verify that everything is working correctly copy the required support files and folders from ~/Music/iTunes to ExternalVolumeName/Compressed Music.
Resetting your lossless library and retaining the playlist structure will take more work because to the best of my knowledge iTunes will not allow you to mass import files from a hard drive except upon initial installation of the software when you are asked if you wish to allow iTunes to scan your attached drives for music files. As you have already found that simply changing the preference location does not seem to work any longer further complicates the matter.
If I get any inspired ideas I will definitely let you know. Otherwise, hopefully some of the other frequent fliers will chime in for you, but with Saturnalia fast approaching the boards are probably not going to be too frequently perused for the next few days. I know that Philmco is real good at dealing with the more esoteric uses for iTunes, but I have not seen many posts by him as of late.
Posted 12 May 2008 - 11:11 PM
Someone reminded me of this thread today, and I thought I'd come back and share what I learned. First off, as far as the original problem - that of ituns not being able to spot things, and me having to go through and point them out individually, song by song - I never really got past that. I spent about three hours in front of my computer getting carpal tunnel, pointing it to the proper location of all the songs. Once I got everything happy again, with no little exclamation points, I did some major organizing. I went through and took all the greatest hits and albums that got put into "complilations" when they really didn't belong there, and I moved them were they needed to be. I went through and consolidated the various spellings of band names into one folder (ie. Billy Bragg and Wilco on Mermaid Ave. 1, but Billy Bragg & Wilco on Mermaid Ave 2, which itunes put in two different folders, of course. 6 albums by Steven Ray Vaughn and Double Trouble had no less than 4 folders!). I cleaned everything up and made it as logical as I possibly could, then I threw out my old AAC music folders, and set my Mac Pro to convert all 7,800 songs from lossless to AAC, and place them in a new music folder. That went faster than I expected, but still took around 24 hours. When it was done, I had two folders in my itunes music file, exactly the same, but one lossless, at around 250 gb in size, and one compressed, at about 48 gb in size. From there, all I had to do was figure out how to make iTunes switch from one to another, without getting confused. (and sadly, this means that the next time I plugged in my ipod, iTunes would only let me sync to the new library by erasing and starting over, which meant that I lost all the ratings and playlist data on the ipod... sad since I'd "discovered" many songs I didn't know while traveling through Europe over six months and listening to the ipod a lot. Now I have to/get to rediscover them again. Eventually.)
Essentially, the issue is in that while iTunes 7 wil let you create multiple libraries, which is what I needed all along, it will not let those libraries have different preference settings. What I really needed, was a way to make two identical libraries (in terms of folder structure) but with different preference settings - one set to import apple lossless, and point towards a folder in my itunes music called "lossless", and one set to import in AAC compression, point to a folder in my itunes music called "aac". After a great deal of research, it seems that Apple does not provide a way to do this. The good news is that others do.
I found a few different programs: iTunes Library Manager, Libra, and Multitunes. None of them work quite as seemlessly as I think they should, but all will let you run different music libraries with different preference settings, which is what I needed. In the end, I have been using multitunes, which works inside of itunes, and simply adds an option in the advanced drop down menu to switch libraries. Selecting that option will close itunes, then reopen it with the new library and preferences. This still isn't a perfect solution... For one thing, everytime I buy a new CD, I basically have to import it twice, once as the lossless file, then switching libraries, and importing as a compressed file. There are other ways to do that (convering the lossless, dragging, etc) but none of them are convenient. The bigger annoyance is that there are two completely different sets of user data on the two libraries, in terms of song ratings, most played, last played, etc. One set which reflects what I've been listening to through the main system, the other reflecting what I listen to on my ipod (though I should point out that they started from the same place - where I converted the lossless library to the AAC.. they've just been diverging ever since, and will contineue to do so). In the ideal world, that data would be shared, and when I rate the new song by White Stripes a 5 on my main system, the ipod would know I liked it as well, but because of the way this stuff works, that simply isn't going to happen... at least not until Apple recognizes the need to let their multiple libraries carry different preference settings (and probably not even then, as it would probably be a pain sorting out which part of those preferences files they could share, and which ones they couldn't.)
So, that's how I'm running my system now. I wish it were prettier, but at least it's running smoothly and iTunes is able to find everything again. As far as I can tell, even if some things go wacky, I shouldn't have to go through and manually point songs ever again.
Hopefully this will help some others if they get into a similar fix, and maybe spur Apple to think a bit more about how people are managing their massive libraries of music.