Database is incomplete, how to debug? - Printable Version +- cool tech zone zone (https://forum.cooltech.zone) +-- Forum: Tangara (https://forum.cooltech.zone/forumdisplay.php?fid=3) +--- Forum: General (https://forum.cooltech.zone/forumdisplay.php?fid=5) +--- Thread: Database is incomplete, how to debug? (/showthread.php?tid=79) |
Database is incomplete, how to debug? - js - 2024-12-19 Hi, I received my Tangara yesterday, and have been playing around with it. First I flashed the 1.0.1 firmware, then I copied my moderately large library over. Tangara created its database automatically. However, I noticed the database is missing several tracks/albums. I can listen to those through the file browser feature, but they don't show up in the indexed views. My best guess is that one of my tracks has a tag that Tangara doesn't like, and it aborts the database scan early? I have no direct proof, but the albums I noticed are missing are ones with later file creation dates (newer additions to the library) so I suspect they would be scanned last. It's possible there are older albums also missing, but I haven't noticed any yet. I manage all my music through beets so I think all my tags should be more or less consistent. I'm trying to debug this issue, but I'm not sure how to get the data I need about which tracks are missing. What is the best way to inspect the Tangara database contents? I might be able to do a diff with my beets track list and figure out where it's choking, if my theory is correct. My LOG and LOG.old files are both empty - 0 bytes. I tried getting the info through the Lua console in Tangara Desktop Companion but anything more complex than a one-liner gives me Command returned non-zero error code: 0x1 (ERROR). Even running t = true in the console gives this reply. My database schema version, according to Tangara, is 8. Would love some help debugging this! Thanks! - JS PS: very minor, but my Tangara Companion is showing the database file size incorrectly as X GiB instead of X KiB. RE: Database is incomplete, how to debug? - jacqueline - 2024-12-20 Hi! So first question: is your music just in the root of your SD card, or is it in a Music/ directory? The next firmware release (coming soon later today!) will help with tracks not in a Music/ directory not being picked up. Otherwise, if you keep Tangara plugged in + the serial port it exposes open during indexing, it should log any tag parsing errors it encounters whilst indexing your tracks. These could be useful for tracking down why some of your tracks are showing up but others aren't. On your PS, this should be fixed at head I think: https://github.com/haileys/tangara-companion/commit/f87b2350eaf81be9ed263a95e800275ddfe2b29b RE: Database is incomplete, how to debug? - js - 2024-12-20 (2024-12-20, 12:09 AM)jacqueline Wrote: Hi! Hi jacqueline! Thanks for getting back to me so fast. My music is one folder per album inside a "/music" (lowercase) directory. I have just started a database update with the serial port open. So far the only logs are related to missing TTS items (a feature I was thinking I might have to develop, but it's 1000% awesome it already exists!!) however the scan has just started so I will leave it running and report back if I find anything useful. I also very briefly played around with the Lua repl over usb serial and it seemed to work better than my experience with the Lua console in the desktop app. So that's now a viable fallback option for investigation if I don't see anything interesting come up in the logs. On top of being an awesome consumer device to begin with, it is legitimately SO COOL that I'm able to do this with Tangara. It's really something special, and I'm so lucky to have one. Thanks for creating it! RE: Database is incomplete, how to debug? - js - 2024-12-20 I've done some more troubleshooting, but unfortunately still haven't found a solution. The first thing I did was double check the tags on some of the files I noticed aren't showing up in my database. They look fine to me. They also show up on my phone just fine, so I don't think it's a tagging issue. Then I ran the database update with the serial console open like you suggested. I only saw 2 things: 1. A bunch of warnings trying to parse cover.jpg in every folder -- which I guess is to be expected since those aren't music tracks. 2. Some warnings of a hash collision. This is also not unexpected, since I have 3 albums in different folders that are, to MusicBrainz, the "same album." I removed the offending hash collision files and re-scanned but no new logs appeared beyond the cover.jpg ones, and the tracks are still missing. I tried running the following in the serial Lua repl, to get a dump of all album titles in the database. I'd like to at least narrow down how many / which albums are missing other than the ones I happened to manually check: Code: do But it crashed Tangara (out of memory?) Code: Guru Meditation Error: Core 1 panic'ed (Unhandled debug exception). Then I decided to try to take a look at the .tangara-db directory contents. Mine looks like this: Code: .rwxr-xr-x 153 js 31 Dec 1979 000005.ldb I ran cat *.ldb | strings and searched for the missing track/album data, and they're in there! It has the full file paths of the tracks, the album titles, and the song titles. So, I'm still not sure why they're not appearing in the All Albums, All Tracks, or Albums by Artist lists. I'm open to other suggestions about what to try next. Later tonight, I am planning to wipe my SD card and start fresh, with a smaller subset of my library including these albums to see if they show up then. If that still fails, I'll try converting all the file formats to mp3 or ogg and see if that changes anything. RE: Database is incomplete, how to debug? - ailurux - 2024-12-20 Hi, thanks for the detailed info, this really helps. We've had a few similar reports of database indexing not working for tracks it should work with, but this is the first I've seen of the database having the entries and it not showing up, that's very odd. Would you mind sharing the entire database as a zip, or however is easiest for you? It might shed some light on these issues to see what entries are there and if there's any reason they shouldn't show up on the device. The repl error implies a stack overflow, which isn't uncommon for the repl as it runs with less stack available than the lua in the firmware. I tried it myself and got the same issue, though the albums do show correctly for me on the device, so it's unlikely to be the cause itself. RE: Database is incomplete, how to debug? - js - 2024-12-20 I tried this again with a minimal library containing one known working album and one known not working album. I backed up then deleted the .tangara-db folder for my full library, cleared out my /music directory except for two albums, and generated a new database with the new smaller library. The same issue is still happening. The two albums I kept are Angel Olsen's "My Woman" (working) and Magdalena Bay's "Imaginal Disk" (not working). I was able to run the Lua script successfully this time, and the only album name it printed was "My Woman". But once again, inspecting the database strings shows Imaginal Disk tracks, paths, and the Magdalena Bay artist. I'm attaching screenshots from Tangara of the various views as well as the new minimal database as a zip file. I am still on the 1.0.1 firmware. Here is the database zip: db.zip (Size: 5.34 KB / Downloads: 1) The files view, with both folders present: The All Albums view, with only My Woman present: The Albums by Artist view, with only Angel Olsen present: RE: Database is incomplete, how to debug? - ailurux - 2024-12-21 Ok I had a look through the database you sent me (thanks!), and it seems that it might be an issue with the tagging for those tracks. There doesn't seem to be album or artist tags that we're able to read for those ones, so they don't show up in the album or artist views. I can see the database does know about those tracks though, can you check if they show up in "All Tracks"? I noticed that the filepaths for these tracks do contain the album and artist, and in future we want to add better support for tagging tracks based on common filepath structures (ie, we can make an educated guess for artist and album if the filepath looks like "Artist/Album/01 Track.flac" or "Artist - Album/01 Track.flac"). But at the moment for a track to show up in the album or artist indexes, we rely on tags in the file itself. RE: Database is incomplete, how to debug? - js - 2024-12-21 They don't appear in All Tracks either. I've checked the tags and they really look okay to me, but maybe there's something subtle I'm missing. Here's the ffprobe for Magdalena Bay's "She Looked Like Me!" - it appears to have proper Artist and Title tags as far as I can tell. Code: ffprobe version n7.0.1 Copyright (c) 2007-2024 the FFmpeg developers Here's the ffprobe for Angel Olsen's "Intern": Code: ffprobe version n7.0.1 Copyright (c) 2007-2024 the FFmpeg developers I diffed the two outputs, and obviously the contents of the tags are not shared between the two - but the only potentially notable differences I noticed are that Angel Olsen has a "COMPOSER" and "COMPOSERSORT" field which Magdalena Bay doesn't. Same for "GENRE." Magdalena Bay has "LYRICS" that Angel Olsen doesn't. I tried removing that tag and it made no difference to Tangara. RE: Database is incomplete, how to debug? - chrisw_b - 2024-12-21 Oh I think I've come across a similar issue. Funnily enough I had no problems with that Magdelena Bay album, nor anything else I had saved locally as a flac from Bandcamp. However, an album that I (sadly) bought off amazon, which came as MP3 files, for some reason did not show up when imported to the tangara with the `beet convert -d /PATH/TO/TANGARA/music` command, and the recommended ``` Code: convert: The files did show up fine when I added `never_convert_lossy_files: true`, and they were simply copied as mp3s instead of converted to opus files, which is a good enough solution for me, so no real complaint here Here's the output of one of the original MP3 files from Gapless Code: /run/user/1000/doc/ea77cd6b/01 Over You.mp3 Hopefully there's some overlap there and it helps track down what's going on with the database, but let me know if there's more info that might be helpful! RE: Database is incomplete, how to debug? - js - 2024-12-21 (2024-12-21, 05:43 PM)chrisw_b Wrote: Funnily enough I had no problems with that Magdelena Bay album, nor anything else I had saved locally as a flac from Bandcamp. Huh, that's weird. Because both the albums I chose were downloaded as flac from Bandcamp. The working Angel Olsen album was downloaded several years ago vs. the not-working Magdalena Bay album which was downloaded this year. Don't know if that helps or just causes more confusion! RE: Database is incomplete, how to debug? - ailurux - 2024-12-21 This is really helpful, thanks! I vaguely recall we were having some issues with tracks with LYRICS tags a while back but I believe that's since been fixed, and you removed the tag anyway so it's probably not that. Oh! I think I might have a idea. I noticed the track that failed to index has no genre tag. We fixed one bug relating to this already, where if you have no genre tag and you have no specific media type directories (ie, "Music/", "Audiobooks/", "Podcasts/"), then we just assume it's music. *But* I've just reviewed it and this introduced a new bug where if you *do* have a Music/ directory and there's no genre tags, it doesn't assume it's music. I'll fix that this week. Oops. If that is the cause in this case, you should be able to fix it by either: - putting all tracks in the top level and deleting the "Music/" folder (as well as any other media type folders) - adding a genre tag (literally anything would work so long as it's not empty) Also noticed another issue that "Music/" is indeed case sensitive here. So it's actually more likely related to that, because it checks that before checking the genre tag. I'll also fix that this week, it shouldn't be case sensitive. RE: Database is incomplete, how to debug? - js - 2024-12-21 (2024-12-21, 09:37 PM)ailurux Wrote: Also noticed another issue that "Music/" is indeed case sensitive here. So it's actually more likely related to that, because it checks that before checking the genre tag. This worked for me with the "small 2 album library"! Renaming the directory from "music" to "Music" has the tracks showing up as expected now. I will try the same with the full library, but it will take much longer to scan, so I won't be able to provide a quick update. But this does seem like the easiest fix at the moment. RE: Database is incomplete, how to debug? - chrisw_b - 2024-12-22 (2024-12-21, 09:37 PM)ailurux Wrote: Also noticed another issue that "Music/" is indeed case sensitive here. So it's actually more likely related to that, because it checks that before checking the genre tag. whoops! I'm too used to making directories lowercase on case sensitive systems (...and also couldn't remember/didn't check if exFAT was case sensitive) RE: Database is incomplete, how to debug? - jacqueline - 2024-12-29 exFAT is case insensitive, but case preserving. It's annoying. We've fixed this at head now, so v1.1 at the end of this week should no longer have issues with media directories that have unexpected cases. |