The process requires downloading aac source and then doing aac->wav->mp3 conversion. I got the required commands and put them in a bash script:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
url="http://www.ambientmusicguide.com/audio/cosmiclounge/"; filePrefix="The_Cosmic_Lounge_0"; fileSuffix="_hosted_by_Mike_G"; for i in `seq 13 32`; do filename=$filePrefix$i$fileSuffix; aacSource=$filename".aac"; wavSource=$filename".wav"; wget $url$aacSource; faad $aacSource; rm $aacSource; lame -h -b 100 $wavSource "${wavSource%.wav}.mp3"; rm $wavSource; done |
Even if you do not know bash, that code is simple enough to understand how slow the process is. aac->wav->mp3 (using faad and lame commands) takes about 8 minutes. That means downloader (wget) has to wait 8 minutes every time before it starts downloading next episode. Thats 8 * 32 = 4 hours of wasted time!! So how can I avoid it? By making sure wget downloader never has to wait for conversion process to complete. To achieve that, I have to start a new thread for the conversion. Essentially this means that I will "detach" lines 10-15 from the overall process tree so that they are called independently and are not waited to be finished. Separating the code is simple enough by putting it in a function (and pass required file names as a parameter so it knows what file to work on at any given time). But how does bash know to start a new thread and not to wait for the function to finish before moving on? By putting a & instead of the usual ; at the end of the call :D
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
url="http://www.ambientmusicguide.com/audio/cosmiclounge/"; filePrefix="The_Cosmic_Lounge_0"; fileSuffix="_hosted_by_Mike_G"; function audioConvert { aacSource=$1; wavSource=$2; faad $aacSource; rm $aacSource; lame -h -b 100 $wavSource "${wavSource%.wav}.mp3"; rm $wavSource; exit; } for i in `seq 14 32`; do filename=$filePrefix$i$fileSuffix; aacSource=$filename".aac"; wavSource=$filename".wav"; wget $url$aacSource; audioConvert $aacSource $wavSource & done |
As much as ; vs & is most likely typical for seasoned local app developers, coming from a web background this technique really fascinates me. So the result is:
- download begins
- while conversion takes place next file is being downloaded already
- if second download finishes before first file is done converting it will start a third download session, start conversion of second file while continuing working on the first (does not happen often since my download speed is slower than the conversion process)
My next concern was whether it would pile up the threads on top of each other on only one of my cores slowing down the conversion process. As it turns out, linux kernel is smart enough to allocate every new thread to the next available idle core. So if we have a third case scenario occur then second conversion process will be started on a different core so now I have 2 cores going at 100% (out of 8 available so I do not even notice the computer is doing something!) thus I have a truly simultaneous conversion going on and not one interrupting the other as single core would have done. You can see this in action by looking at CPU usage graph.
Only one little issue is that I did not bother to forward output from each thread to a different terminal session, so I have a bunch of gibberish in the console as I have 2 or sometimes 3 commands fighting to display their output.
