3
\$\begingroup\$

I created a way to export my Google Play playlists to a text file and also a way to export all of my songs in Google Play to a text file. Now I have both the song list (in an array format) and playlist data (in object and array format) in their respective text files. Here is a simplified version of the files:

[
    "song 1 name",
    "song 2 name",
    "song 3 name"
]

and

{
    "playlist 1": [
        "song 1",
        "song 3"
    ],
    "playlist 2": [
        "song 1"
    ]
}

When I run them in my script, it works correctly and creates the following output:

{
    "song 1": [
        "playlist 1",
        "playlist 2"
    ],
    "song 2": [],
    "song 3": [
        "playlist 1"
    ]
}

Here's my code:

var songInput = document.getElementById("songInput"),
    playlistInput = document.getElementById("playlistInput"),
    result = document.querySelector("textarea");

var songList, 
    playlistData;

function loadData(id, elem) {
    if(elem.files
    && elem.files[0]) {
        let myFile = elem.files[0];
        let reader = new FileReader();
    
        reader.addEventListener('load', function (e) {
            if(id === "songInput")
                songList = JSON.parse(e.target.result);

            if(id === "playlistInput")
                playlistData = JSON.parse(e.target.result);

            checkBothAdded();
        });
    
        reader.readAsBinaryString(myFile);
    }
}

function checkBothAdded() {
    if(songList
    && playlistData) {
        getSongPlaylistData();
    }
}

var songData = {};
function getSongPlaylistData() {
    for(let song of songList) {
        let playlistsItsIn = [];

        Object.keys(playlistData).forEach(function(key) {
            if(playlistData[key].includes(song)) {
                playlistsItsIn.push(key);
            }
        });

        songData[song] = playlistsItsIn;
    }

    result.value = JSON.stringify(songData, null, '\t');
}

songInput.addEventListener("change", function() {
    loadData("songInput", this);
});

playlistInput.addEventListener("change", function() {
    loadData("playlistInput", this);
});
<label>Song text file (generated from <a href="https://webapps.stackexchange.com/q/108103/140514">this answer</a>): <input type="file" id="songInput"></label>
<br>
<label>Playlist text file (generated from <a href="https://webapps.stackexchange.com/a/106604/140514">this answer</a>): <input type="file" id="playlistInput"></label>
<br>
<textarea style="width: 500px; height: 200px;"></textarea>

It works on this small set of data I hand-wrote, but it freezes the browser page when I run it on my ~4,000 song list and substantial playlist file.

How can I do this same thing while helping it perform better so it doesn't freeze my browser tab?

I'm theorizing that it would work faster if I keep them as text files and try to parse each of the playlist names if the song is found, but I'm curious if I can speed up/reduce the memory of this object approach somehow.

\$\endgroup\$
2
  • \$\begingroup\$ Without looking at your code, to prevent blocking code, Two options, run parallel by using a webWorker to process the data. OR time share using setTimeout to give the browser some of the CPU time while you are crunching the numbers.Data is processed so much faster when you are watching a progress bar, rather than waiting on a hung page. \$\endgroup\$
    – Blindman67
    Feb 17, 2019 at 18:11
  • \$\begingroup\$ @Blindman67 Thanks, I know those are ways to work around the issue, but I am searching more so how to increase the speed/reduce the memory of the JavaScript object approach (though, as the answer I posted says, I'm just going to use the regex search version I made for now). \$\endgroup\$ Feb 17, 2019 at 22:21

2 Answers 2

3
\$\begingroup\$

Use Map for fast lookups

The main performance cost is the search for songs

The best way to get better lookup performance is to use a Map to store the songs. Maps use a hash to lookup items and is very fast once the Map has been created.

You can create a map as follows, it adds an array for the playlist.

const songMap = new Map(songList.map(name => [name,{name,playLists:[]]));

Then you iterate the playlist and the songs it contains, adding the playlist to each song

for (const [playListName, songs] of Object.entries(playlists)) {
    for (const songName of songs) {
        const songItem = songMap.get(songName); 
        if (songItem) {
             songItem.playLists.push(playListName);
        }
    }
}

You can then iterate the songMap and create the final object containing songNames with arrays of playlists

 const songPlaylists = {};     
 for(const song of songMap.values()) {
       songPlaylists[song.name] = song.playLists;
 }

Then convert to JSON if you need.

Why the processed store?

However I would argue that the whole process is pointless unless you are extracting some higher level information from this process.

Ideally you would store the songs in a Map and each playlist also stored in a Map and would contain a Map of songs. Thus you can lookup playlists a song is in as follows

  function findSongPlaylists(songName) {
      const playlists = [];
      for (const playlist of playlistMap.values()) {
          if (playlist.songs.has(songName)) { playLists.push(playList.name) }
      }
      return playlists;
  }

To further improve the store, you would assign a unique id (a number) to each song and playlist and use the and do all the searches using ids. This will give even better performance and save a huge amount of RAM.

For casual viewing it is the ideal.

BTW the link you provided showing how you extracted the song data has a very long way of getting the data. All that data is store in the page indexed DB and can be extracted and saved to disk with a few lines of JS. (no need to scroll the page). Use Dev tools > application tab and open indexedDB to find the data.

Update Extra

Looking through the link you posted Copy Google Play Music songs all the solutions required rather hacky method (due to song list being a view) of slowly scrolling over the data to acquire it. The question is locked thus I am unable to add a better answer. As this question is related I will add a better way of sourcing the data here.

Paste the following snippet into the Dev-Tools console and hit enter. It will extract all the songs and save them to your download directory as a JSON file (named music_[your id].tracks.json)

The json file contains (data refers to the parsed json file)

  • data.fields an array of field names in the same order as in the array
  • data.tracks an array of records (one per song) containing fields in the same order as in data.fields.

Note I do not use Google Music (Used a friends account so had limited time) and hence I guessed the content. You can add extra fields if you find the array index. To get a single record go to dev-tools > application (tab) > indexedDB > music_[your id] and select a record.

Example record extracted from music_####.tracks For security ids have been masked with # or A...

"########-####-####-####-############":["########-####-####-####-############","Sheep","https://lh4.ggpht.com/AAAAAAA-AAAAAAAAAAAAAA_AAAAAA...","Pink Floyd","Animals","Pink Floyd","sheep","pink floyd","animals","pink floyd","Roger Waters","Rock",null,620059,4,0,0,0,1977,0,null,null,0,0,1372528857298236,1408583771004924,null,"AAAAAA...","AAAAAAAA...",2,"",null,"AAAAAAA...","AAAAAAAA...",128,1372582779985000,"https://lh3.googleusercontent.com/AAAAAA_AAAA...",null,null,[],null,null,null,null,null,null,null,null,null,null,"AAAAAA...",[],null,null,null,null,0,null,null,[["https://lh3.googleusercontent.com/AAAAA...",0,2],["https://lh3.googleusercontent.com/AAAAA...",0,1]],true,6,null,1],

To keep the file size down the data is in an array. Nulls have been converted to 0.

(function () {
    console.log("From https://codereview.stackexchange.com/a/213683/120556"); // DO NOT REMOVE THIS LINE if you publish this snippet.
    console.log("Just a moment locating music DB...");
    const data = {
        dbName:"",
        info: "From Google Play Music App IndexedDB.\nField names in order of song array index.\nTo keep JSON size down null has been replaced with 0.\nDates ar in micro seconds 1/1,000,000 sec (I don't know what date1, date2, date3 represent)\nImages may require additional query properties.\nSource https://codereview.stackexchange.com/a/213683/120556",
        fields: "GUID,song,image1,band,album,albumArt,composer,genre,length,track,tracks,disk,disks,year,plays,date1,date2,bitrate,date3,image2".split(","),
        tracks: [],
    };
    const idxs = [0,1,2,3,4,5,10,11,13,14,15,16,17,18,22,24,25,34,35,36]; // idx of info to extract. There are many more field, 
    indexedDB.databases().then(info => {
        const name = data.dbName = info.find(db => db.name.indexOf("music_") === 0).name;
        indexedDB.open(name).onsuccess = e => {
            console.log("Extracting tracks fro DB " + name);
            const t = e.target.result.transaction("tracks", IDBTransaction.READ_ONLY); 
            t.oncomplete = () => {
                Object.assign(document.createElement("a"), { download : name + ".tracks.json", href: URL.createObjectURL(new Blob([JSON.stringify(data)]), {type: "text/json" })})
                    .dispatchEvent(new MouseEvent("click", {view: window, bubbles: true, cancelable: true}));
            }
            t.objectStore("tracks").openCursor().onsuccess = e => {
                if (e = e.target.result) { 
                    Object.values(JSON.parse(e.value)).forEach(t => data.tracks.push(idxs.map(i=>t[i]===null?0:t[i])));
                    e.continue();
                }
            }
        }
    }).catch(()=>console.log("Sorry can not complete data dump :("));
})()

To extract a record from the JSON

// data is the parsed JSON file
const songName = "A song name";
const songFieldIdx = data.fields.indexOf("song");
const track = data.tracks.find(rec => rec[songFieldIdx] === songName);

To convert track array record to track object

const trackObj = {}
for(const field of data.fields) { trackObj[field] = track.shift() }
\$\endgroup\$
5
  • \$\begingroup\$ Thanks for the input! Maps are a good way to go but I haven't really used them before so this is a good step for me. I can't seem to get the indexedDB to give me any readable data. Any idea how to work with it? \$\endgroup\$ Feb 18, 2019 at 4:28
  • \$\begingroup\$ @ZachSaucier I have updated my answer with info on how to dump indexedDB to your download directory and then basic usage. NOTE if you have more than one Google Music account on the computer the snippet will get the first one. You will need to modify it to extract the one you want. \$\endgroup\$
    – Blindman67
    Feb 18, 2019 at 18:38
  • \$\begingroup\$ I'm trying to find playlist information using this DB approach but can't seem to find any. Any idea where that might be stored? I suppose it could just be on Google Play's server side \$\endgroup\$ Feb 23, 2019 at 17:59
  • \$\begingroup\$ @ZachSaucier if its not in the DB then likely a XHR request. Open Dev tools and go to the network tab. On main Google Music select a play list, back on dev tools, network if there is any data exchange it will show up there. It will likely contain index references to songs rather then song names. \$\endgroup\$
    – Blindman67
    Feb 24, 2019 at 1:31
  • \$\begingroup\$ After looking at the Networks tab, it doesn't appear to be through an XHR request. The only potentially relevant XHR request is the same no matter what the playlist is. The songs are also loaded one by one so I don't think that'd make much sense to do over the network \$\endgroup\$ Feb 24, 2019 at 2:15
0
\$\begingroup\$

I am still interested if anyone knows how to improve the speed of the JavaScript object approach, but I ended up searching the playlist text file with regex instead. It now runs in only a few seconds (without needing a background worker or anything):

var songInput = document.getElementById("songInput"),
	playlistInput = document.getElementById("playlistInput"),
	result = document.querySelector("textarea");

var songList, 
	playlistData;

function loadData(id, elem) {
	if(elem.files
	&& elem.files[0]) {
		let myFile = elem.files[0];
		let reader = new FileReader();
    
    	reader.addEventListener('load', function (e) {
    		if(id === "songInput")
    			songList = JSON.parse(e.target.result);

    		if(id === "playlistInput")
    			playlistData = e.target.result;

    		checkBothAdded();
    	});
    
    	reader.readAsBinaryString(myFile);
	}
}

function checkBothAdded() {
	if(songList
	&& playlistData) {
		getSongPlaylistData();
	}
}

/* Needed to escape song names that have (), [], etc. */
function regexEscape(s) {
    return s.replace(/[-\/\\^$*+?.()|[\]{}]/g, '\\$&');
};

var songData = {};
function getSongPlaylistData() {
	for(let song of songList) {
		let playlistsItsIn = [];

        /* Use regex to search the playlist file for all instances,
           add the playlist name that it's in (if any) to an array. */
        let reg = new RegExp('"([^"]*?)":(?=[^\\]]+' + regexEscape(song) + ')', "g");
        let result;
        while((result = reg.exec(playlistData)) !== null) {
            playlistsItsIn.push(result[1]);
        }
        
        // Update our song data object
		songData[song] = playlistsItsIn;
	}

	result.value = JSON.stringify(songData, null, '\t');
}

songInput.addEventListener("change", function() {
	loadData("songInput", this);
});

playlistInput.addEventListener("change", function() {
	loadData("playlistInput", this);
});
<p>Please note that this will take a few seconds to run if you have a large amount of songs and playlists.</p>
<label>Song text file (generated from <a href="https://webapps.stackexchange.com/q/108103/140514">this answer</a>): <input type="file" id="songInput"></label>
<br>
<label>Playlist text file (generated from <a href="https://webapps.stackexchange.com/a/106604/140514">this answer</a>): <input type="file" id="playlistInput"></label>
<br>
<textarea style="width: 500px; height: 200px;"></textarea>

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.