I just completed my latest experiment 🙂 It is a revamped Web version of Orion Sky Lawlor’s “GPU-Accelerated Rendering of Unbounded Nonlinear Iterated Function System Fixed Points” (see here).
For good measure I have added a bit of music interaction 🙂 I suggest you have a look at the live demo here (Chrome recommended). A Youtube recording can be found here.
The music I selected above is by Dexter Britain and licensed under a Creative Commons license. (see www.dexterbritain.co.uk)
Programming background information:
- Starting from Dr. Lawlors opengl/C++/Windows program the main step was to get rid of any opengl legacy fixed pipeline rendering and replace it with a programmable pipeline WEBGL equivalent. Then some WEBGL limitations still had to be dealt with, e.g. adding missing WEBGL shader functions, replacement of unsupported for-loops in shaders or unsupported texture modes, etc. Luckily this process proved easier than expected. The resulting WEBGL/JavaScipt program works quite fluidly in Chome (on my GXT460). Firefox’s WEBGL implementation also works for the most part – but it occasionally it crashes for no good reason and it is rather slow in compiling the shaders.
- Some surprise obstacles came with the attempt to implement the music playback and hook it up WebAudio’s AnalizerNode infrastructure: The first attempt to use some XMLHttpRequest based loading with AudioContext.decodeAudioData() processing proved a total failure, because Firefox’s implementation (versions 31 -33) just fails to decode the (supposedly “invalid”) mp3 data. Also it became obvious that the upfront decoding of the complete mp3 file (in Chrome) adds an unacceptable delay to the music startup. Interestingly playback of the same mp3 file works flawlessly in Chrome as well as in Firefox when using the HTMLAudioElement. So that’s what I am using now. Unfortunately that API is not implemented consistently in the different browsers and particularily the event handling in Firefox leaves a lot to be desired. But even Chrome fails to properly “loop” a song but instead goes into some corrupt state at the end of some (especially longer) songs. Once again browser specific tinkering is called for…
- The page uses multiple AnaylzerNodes with low-/band- and high-pass filters to detect certain music patterns that are then reflected in the graphics. Unfortunately Firefox again proved a difficult customer: Connecting FilterNodes to the HTMLAudioElement here sporadically leads to corrupted music playback. Also the Firefox FilterNodes produce completely different results than the Chrome ones.
Did I mention that I love fractals? In fact I do and that’s why I created the below intro:
I am using orbit trap logic that I implemented in a WEBGL fragment shader. The add-on text scrollers/messages sport an exploding-pixels effect which is done using plain Canvas pixel operations. For good measure I added my WebAudio based SID chiptune playback (see my previous posts).
I guess that next time I’d better leave all the graphics to the GPU .. because my CPU is just getting too old. In case you are using a recent browser and if you have some sort of GPU in your computer, you might try to see the online version by clicking on the above image.
On my second trip to the world of THREE.js and WEBGL I tried out yet another bunch of “new” things:
- use of THREE.js EffectComposers and Shaders to create “glow” effects, etc
- creating lightweight 3D text rendering by rendering a 2D font into a texture and by creating a geometry that then represents each character as two simple textured triangles
- create THREE.ShaderMaterials (i.e. WEBGL vertex and fragement shaders) to then apply visual effects to the above 3D text (e.g. rolling the characters or exploding the text-page, etc)
Then I came across Felix Woitzel’s fascinating “Travelling Wavefronts” demo and I had to understand how its WEBGL shader logic is done 🙂 After some tinkering I ended up with a slightly cleaned up version of my own with some added visual effect features:
After throwing all of the above together and and adding a bit of WebAudio based SID chiptune playback (see my previous experiment), I ended up with a little Web intro 🙂 Those with a reasonably fast GPU may see it in action here (due to my old CPU I really like to use as much of the GPU’s power as I can..).
At last Firefox and Chrome seem to be making steps forward with regard to audio 🙂 .. giving me an opportunity to migrate my Flash based music player (see previous experiment) to an HTML5-only implementation.
PS: Unfortunately Chrome and Firefox still don’t seem to be on the same page with regard to correct chaining of Nodes.. still some work to be done.
By chance I had recently come across a very cool 3D web page and thereby discovered what modern web browsers meanwhile are capable of.
The experiment here is my THREE.js based equivalent of “hello world” – which I did to just get a first glimps at what it means to do 3D in the browser (the live version is here).
I added a Web Worker based fractal generator to experiment a bit with the benefits of a respective asynchronous approach.
Eventhough they haven’t managed to get the tranparency handling right yet (see here), the guys at THREE.js are certainly doing a great job abstracting from the ugly details of WEBGL wiring! (What they don’t seem to have any clue about is API management and backward compatibility.. e.g. repeatedly inverting the orientation of their 3D coordinate system from one version to the next.. LOL)
Having the building blocks of my SID player available (see earlier posts), it really was no brainer to take up the suggestion of a colleague to take his Rockbox version of a MOD player and also turn it into a Flash version (it took about half an hour).
To try out at least something new, I decided to tap the WebService of a popular chiptune page and learn how that is done with PHP. Also I am using AJAX to integrate the resulting data.
As I had already done in my SID version I am using a PHP/cURL based proxy script to work around the Flash security restrictions and load the music files indirectly from the respective domain.
The result can be seen here.
This next little experiment led me into the world of Joomla 2.59 (an open source CMS system). Since PHP and MySQL are among the few things that my web hoster is comfortable with, there wasn’t much of a choice really. Still I jumped at the opportunity to get some hands-on PHP5 experience.
As expected a simple Joomla site was quickly setup with pre-existing building blocks like user management, commenting functionality and a simple release system to share my source code. Also it came as no surprise that the ‘point & click’ based Joomla software development approach got hairy once I needed my own custom functionalities. With hairy I mean the energy that is wasted to find the different places (e.g. via plugins, modules and component config or site templates) that need patching to rectify the “incorrect turns” that the “framework” is otherwise taking (or to just add missing hooks).
To make the music player from my previous experiment a bit easier to use I decided to put it into its own floating window and integrate it with some HTML based playlist functionality. After some failed attempts to get my floating window re-positioning code to consistently work in different web browsers (using “plain HTML”), I ended up using jQuery which solved the respective problems just fine. jQuery then also proved useful for the CSS based highlight/expand/collapse functionalities that I added and later it was the base for implementing a dynatree.js based file browser with “load on demand” and “drag & drop” functionality.
The final result (here) turned out quite as I wanted.
Still I was somewhat surprised about the “immaturities” that sometimes manifested in the above tools. With that I mean that eventhough I used these tools superficially at best I already ran into a couple of bugs which previously had gone undetected – which does not speak for thourough testing. However it was a pleasure to then witness the speediness with which these bugs got fixed once they were reported (I hope they also created a new testcase.. ). Community driven software development at its best 🙂
When it appeared in the early 80s, the C64 home computer was appreciated particularily for the capabilities of its SID (Sound Interface Device) audio chip.
The SID audio chip featured tree channels, each with its ADSR envelope generator, ring modulation and filter capabilities. Technically it was a small revolution for the home computer domain of the time.
In the more than 30 years since the C64 first hit the shops, countless musicians have created tens of thousands tunes with that characteristic early 80s sythesizer sound (see High Voltage SID Collection). People that grew up with the C64 will certainly still remember some of them.
With the advent of the personal computer people started to program C64 emulators and by now there is quite a selection of more or less complex ones available for different platforms.. so far (i.e. when this experiment was done in 2012) none of these was available for the web browser.
Goal of the experiment: Create a lightweight C64 emulator that allows to playback original “music files” within the web browser (i.e. multi-platform). Lightweight meaning that the program should be fast enough to execute well also on computationally weak devices like smartphones. This certainly qualifies as a technically interesting if practically irrelevant ‘just for fun’ endeavor 🙂
The diagram to the right shows what the emulator is about: Contrarry to file formats like .mp3 or .ogg, .sid music files do NOT represent a dedicated audio file format but are in fact little more than arbitrarry C64 program files.
Within those programs any official as well as any undocumented C64 feature may actually be used to influence the SID chip’s audio output. An example of an official feature might be the program’s use of the SID chip’s independent oscillator register while an undocumented feature might be one of the ways to trick the SID chip into playing a specific audio sample:
The playback of audio samples was never an official feature of the SID chip.
The tricks exploited for the respective playback therefore are usually highly timing critical: While regular audio output is handled quite independently within the SID chip, sample playback must be explicitly handled by the program code, i.e. depending on specific execution times of 6510 CPU instructions, the timing of interrupts generated by the CIA timers or the 6667 video controller chip, etc. (Not to speak of timing interferences originating from the interactions between the different chips, e.g. the VIC may block the CPU during certain tasks, etc..)
Approach: Not having any experience in electrical engineering nor owning the real hardware for analysis, I decided to build on someone else’s existing SID emulation. I picked TinySid created by Rainer Sinsch because it is more compact than the more sophisticated competitors. This choice allowed me to get my ‘proof of concept’ going quickly – however I had to realize later that the existing emulation was limited to only parts of the CPU and the SID and none of the other above building blocks… so there was quite a bit of tinkering left for me to be done.
I did my ‘proof of concept’ with Adobe Flash since I had found that I could cross-compile the existing C code using Alchemy (TinySid is written in C.) This would also give me an opportunity to play with Flash (which I had not done before).
With a bit of extra AS3 code for the UI, the Alchemy compiled emulator code was up and running quite quicky. (It took a while though to then locate and fix the gaps in the existing emulation logic.) You might have a look here to see the completed player in action.
I also reached my cross platform goal, since the player runs about anywhere where Flash is available (e.g. even on my old HTC HD2 smartphone – reflashed to Android.. no pun intended).
It was fun doing some AS3 for a change, but coming from a C++ and Java background, the respective Flash development tools leave a lot to be desired. In particular when using Alchemy. It then helps to not make any programming errors.. 😉