Joomla in hindsight

It’s been some years that I had played a bit with Joomla (at the time version 2.5 was “the thing”). Since it was only kind of an learning experience and the resulting pages were of a hobby project nature, I did not spend any money on extensions but used what was available at the time in some “free to use” version. Still I ended up using about half a dozen 3rd party “extensions” (basic “booking” functionality, etc) some of which I had to customize significantly to make them cover my requirements. But the result did what I wanted and it had probably taken less time than if I had programmed everything from scratch myself.

One “doesn’t look a gift horse in the mouth”.. but some of the “Joomla/extensions” code was obviously subpar, and their authors apparently hadn’t even had an idea how to properly overload methods in sub-classes nor had they known the difference between static and non-static methods, etc. .. but I guess you get what you pay for.

I cannot say that I was thrilled by Joomla’s approach to composing pages either .. writing some “article” and putting it into some “menu” structure is easy enough. But having to define separate “modules” elsewhere (e.g. for the included JavaScript files that a specific page might be needing) and then attaching those via tedious/slow admin-GUI “checkbox clicking” to some “menu item” or some placeholder from the site’s template, etc.. Alas, whenever I came back after some month to make just some minor adjustment it always took excessive amounts of time just to remember how those things where actually connected (since much of the stuff comes from the DB it doesn’t help to do a quick text search on the file system to look for something).

Green banana software

Joomla is actively “improved” and there is what looks like a continuous stream of new releases. A look into the respective https://developer.joomla.org/security-centre.html shows that respective releases are not always an improvement and some severe security flaws where actually absent in older versions and then introduced in some “cool new update” (like some “Severity: High” bugs (CVE-2019-10946 that affected versions 3.2.0 through 3.9.4, CVE-2019-9713 that affected versions 3.8.0 through 3.9.3, etc).

Some other software flaws actually go unnoticed for years before they are eventually noticed by the Joomla developers, e.g.  “Severity: High” CVE-2017-9933 affects 1.7.3 – 3.7.2. (I can almost hear the 2016 sales pitch of the Joomla acolytes, “what you are still using 2.5.9? you must upgrade to 3.5 immediately! older versions are such a security risk!”.. haha, very funny..)

The depressing thing though is that (even for the officially supported/current Joomla versions there are no separate security patches. Instead the official advice always is to update to the next version – which supposedly fixes the problem. This means that you cannot get the 3 files that fixed a specific bug separately but instead you may get a 10Mb zip file that introduces a ton of other changes at the same time (the only exception seem to be the EOL fixes here: https://docs.joomla.org/Security_hotfixes_for_Joomla_EOL_versions/de ).

Joomla’s versioning and updating policy is weird .. or rather disturbing. Some kind of automated updating support is available in the admin GUI – except that it is “somewhat limited”:  My first Joomla instance had been using 2.5.4 and the second one 2.5.9, interestingly their admin-GUI tells me that 2.5.4 should be upgraded to 2.5.5 while my 2.5.9 instance happily tells me that no automatic upgrades are possible. So even if I had ever wanted to update to the last 2.5 version (which I think would have been 2.5.29) then even what should be a minor-sub-version update seems to be something too risky to perform automatically.. seriously?

Regarding security

As indicated above, the software quality of the Joomla core is not that great and
there are tons of more or less severe issues that “pop up” in the various versions. I’d say it is prudent to not expect much with regards to Joomla security and select potential projects accordingly.

From the beginning it is probably a good idea to restrict access as far as possible, e.g. by activating the web server’s basic HTTP authentication for the “/administrator” GUI functionalities.

A Joomla instance isn’t suitable for a “never touch a running system” approach and most sites will probably be trapped in the “update to the very latest version” hamster wheel, thereby volunteering as beta tester for whatever green bananas Joomla wants to field test. (You better not use any 3rd party “extension modules” unless you are absolutely confident that the respective provider will still be there tomorrow to get you an updated version for the next Joomla version – or else you’ll end up rewriting those portions of your site.)

Personally I chose the different approach of just back-merging the code changes for the “Severity: high” Joomla fixes into my old code base. Thus avoiding to find replacements for the long gone “extensions” still available for my old version. (This is of course an absolute  no-go and I am most certainly a risk for the Internet and maybe for world peace as well…)

Green banana software meets planned obsolescence

It never fails to amaze me how PHP could ever grow such a large following: The poor design decisions taken in early “versions” are so obvious that even newer versions (thankfully) start to reverse them (see “backward incompatible changes”).

But hey, everybody has the right to design a crappy programming language and then learn from his mistakes. The problem with this crappy language is that it comes with an expiration date: “Each release branch of PHP is fully supported for two years from its initial stable release“.

Like a light-bulb that wants to be replaced after 1000h of use. Only here it works even better.. no need to be broken, let’s replace every 24 month. Add some “backward incompatible changes” and you have a printing machine for money/extra work.
So much wasted opportunities.. just imagine “ANSI C is end of life and all the old programs must be ported to Java8 by the end of the month!”.. splendid, why did nobody think of that one earlier?

So it happens that my hoster informed me that “he will no longer be hosting PHP5 by the end of the month and would I please migrate everything to PHP7”. But of course! I had no plans for the weekend anyway, f*** you very much!

Obviously Joomla 2.5 could not know about PHP7 yet and the Joomla support doesn’t want anybody to use those old very dangerous legacy versions anyway. (Support in the Joomla universe means: getting help when migration to the new version went south.)

Spoiler: In spite of the “Joomla support” propaganda – old Joomla 2.5 (with the manually added security patches) can be “easily” ported to PHP7.

  1. Search for “->$” to find the following indirect variables usage pattern: change “$a->$c[$b]” to “$a->{$c[$b]}” in order to preserve the original semantics in PHP7
  2. Search for “$key = key($this->_observers)” which no longer works here since the foreach loops no longer update the internal state of the array (add “$key++;” within the foreach loops instead) .
  3. Replace preg_replace with respective preg_replace_callback based impls.

After this the Joomla instance will start again and you can go after the deprecation warnings (etc) if you want to cleanup properly.

I do not recommend to use an old version – or any Joomla version for that matter (you saw the Joomla security issue tracker)! but if you are desperate..

PlayMOD

With meanwhile 21 different JavaScript based music emulators in my toolbox it seemed like a logical thing to do..
PlayMOD combines all of my music emulators within one UI to provide online music playback for one of the largest “chiptune and other legacy computer music” collections available on the Internet: The modland.com collection contains about 450’000 such music files (see ftp.modland.com/pub/modules) and the available emulators allow to play more than 98% of these directly within the web browser.

playmod

The UI is based on the design/codebase originally created by JCH for his DeepSID. I made the necessary adjustments for use in the modland.com context and added stuff like the “Visuals”.

sprinkler WiFi update..

ATMEL_TQFP64_200Starting with a little retrospection: So far I had built two sprinkler controllers that I remotely control via a home grown (Java) PC software (see my earlier sprinkler posts).

These devices are based on a cheap ATmega128 micro controller which requires a separate breakout board and a bit of soldering to get going. With some custom PCB this would actually be a neat chip to use – with more IO pins than you’ll probably ever use. But with the breakout board it uses quite a lot of space and it gets very annoying when one of them burns out (e.g. due to some surge on the power grid) and you quickly need to come up with a replacement.

433Mhz-SI4432-Wireless-RF-Transceiver-Module-with.jpg_220x220The devices then use a Si4432 transceiver to communicate with the PC application. The Si4432 theoretically should have an impressive “best case” range and is fairly inexpensive too. As always, solid obstacles (like walls) are the limiting factor for the real range of this transceiver: In my case the obstacles are two tiled roofs and I am ending up with a maximum range of around 50m-70m.

However I soon found, that the Si4432 is pretty fragile (i.e. these things just keep dying for no obvious reason) and most of the really cheap PCBs use 1.27mm pin spacing which makes it a pain in the ass to replace them or to even hook them up to your 2.54mm prototyping boards. I had therefore opted for a somewhat more expensive version with 2.54mm pin spacing. Unfortunately there seem to be different builds (I already saw three) of very similar looking (blue) PCBs, and to make things worse some of these are just incompatible. So when I wanted to just add one more device to my existing zoo I very annoyingly ended up replacing all existing transceiver PCBs because the new version that was sold at that time no longer worked with the older models I already had. Also transmission speed of the Si4432 is not that great (to some degree that may be my fault since I traded in some speed in favour of a more reliable connection).

New approach

wemos-d1-mini-proBased on the above experiences I felt that I should try something new for the 3rd device that I was about to build (something to be used in my greenhouse). I decided to give WiFi a chance after all and replace the ATmega128 & Si4432 with a “Wemos D1 mini Pro” (ESP8266EX based). The idea is that it should be much easier to get some standard WiFi range extender if necessary and the transmission speed should be much higher in any case. The respecive board costs around five bucks (with external antenna & cable) so it won’t hurt if ever there is the need to replace it.

Obviously I will keep the old devices that I already have and I therefore decided to extend my existing software such that it also is capable to deal with WiFi/UDP based communication. I just completed the respective software changes and everything seems to work like a charm (and blazingly fast):

wifi

Meanwhile everything is nicely packed into the device case and ready to go 🙂

RainMakerWiFi

For now just some ESP8266/Wemos D1 related observations. In principle, migrating existing Arduino code to the ESP8266 is pretty straight forward: Most of the existing libraries that I had been using previously, also work on the ESP8266 (I did only have an issue with one specific EEPROM library – which I ended up not using).

But there are significant differences to the “normal” microprocessors I had been using previously:

  • WiFi handling (i.e. the built-in behind the scenes functionality) is the ESP8266’s first priority.. Your application code IS NOT and it MUST NOT get into the way or the device will just crash (there mustn’t be any long running sections in your code.. “yield()” generously – this is the annoying part: having to litter your code with ESP8266 specific yield() code that has absolutely nothing to do with your application logic).
  • The memory model is somewhat more complex than you might initially realize (see http://cholla.mmto.org/esp8266/where.html). Code like interrupt handlers must be in the “correct” area (i.e. RAM) or the device will unexpectedly crash.
  • Power supply is crucial: Insufficient power supply – e.g. via a weak USB connection – may be the cause of unexpected crashes.
  • Lastly the built-in hardware watchdog will also crash the device if not “fed” regularily.

All of the above may lead to “unpredictable” system behavior and tracking down the actual root cause of a problem can be quite an annoying task. Using a tool like https://github.com/littleyoda/EspStackTraceDecoder may then be your best chance to at least get some idea where the problem might come from. (The reference here may also be useful.)

To make things worse I found myself having problems with the IDE I am using (Sloeber 3.0) that may or may not be ESP8266 specific: It may be a general problem of hobbyist IDEs (or Eclipse in particular) but the IDE regularily DID NOT build/upload the correct code. And the ESP8266 oftentimes does not restart properly after a “reset” triggered by the IDE (which may be linked to inadequate USB power supply). I ended up chasing phantom problems that disapeard after a “clean build” and un-plugging of the device. Obviously it is rather annoying when you cannot depend on the correct functioning of your tools!

Regarding the Wemos D1 (ESP8266) hardware: IO pins are scarce and many do have built-in limitations (e.g. they are used to control how the device boots, etc). It seems to be a good idea to use I2C whereever possible to preserve the little that is available. (This may become a pressing issue in case you also intend to use SPI – which may be a prerequisite to hook up things like an SD-card reader.)

Licensing

I had not been concerned with the topic of licensing while I was still in midst of trying to get the basic functionality to work. (Most people probably aren’t while dealing with their Arduino toy projects.) Therefore I had not payed much attention to the licensing conditions of the Arduino libraries that I had been using either.

For a large part respective libraries use LGPL and that license allows you to use some library in pretty much any way you like. BUT…

… there also are libraries that use GPL – which may/should be considered a problem: I for one do not write blanco checks nor do I like the idee of having 3rd parties exploit my work financially without me getting a single dime of the proceeds. As a matter of principle GPL therefore for me is an absolute NO-GO.

Luckily, in the context of Arduino libraries, functionality is usually so small that you can easly write a replacement yourself. And before subjecting your own code to a shitty GPL license you are probably better off writing your own library.

I dumped two respective libraries and replaced them with my own (see https://github.com/wothke/justASK and https://github.com/wothke/TinyLCD). So if ever I release my code it can be CC BY-NC-SA by default and in case anybody wants to make money with it he can get a commercial license).

 

Playing with WebAssembly

I recently noticed that ’emscripten’ meanwhile allows to also generate WebAssembly output. WebAssembly is touted (see e.g. https://hacks.mozilla.org/2017/03/why-webassembly-is-faster-than-asm-js/) to use less space and to run more efficiently (i.e. faster) than previously existing web technologies.. sounds great!

With my portfolio of various ’emscripten’ compiled chiptune music players this seemed like the perfect opportunity to just give it a try (If you want to try this yourself, make sure to really get the latest ’emscripten’ version! Also be warned that the new ‘clang’ version that ’emscripten’ is using now, is more strict with regards to existing C/C++ standards and you may need to fix some of your old bugs in the process..).

Due to the fact that web browsers will load the *.wasm WebAssembly files asynchonously, existing bootstrapping logic may need to be reworked (you’ll need to wait for  the notification of the loaded ’emscripten’ module that it is actually ready – don’t even think about using the SINGLE_FILE hack, it won’t work in Chrome!).

In the case of my chiptune players, migration fortunately wasn’t a big deal (my player was already prepared to deal with asynchronously loaded stuff) and soon I had the first *.wasm results. And from a size perspective, those output files already were good news: In their old asm.js incarnations some of my emulators are rather bulky and in total the size of the nine emulators originally summed up to more than 11MB. The better optimizer used in the new ’emscripten’ already managed to bring those asm.js versions down to about 10MB – but with *.wasm that now shrinks to 5MB. Nice!

I then went about measuring the performance of the different versions (I tested using Chrome 64 and FireFox 57 on a somewhat older 64-bit Win10 machine). I was using my all-in-one “Chiptune Blaster” page as a testbed (see https://www.wothke.ch/blaster/ and https://www.wothke.ch/blasterWASM/). I patched the music player “driver” to measure the time actually spent within the various emulators while they are generating music sample output. I started measuring after each emulator had already returned some sample data (i.e. its program code had already been used) and then measured the CPU-time that it took to generate 10 seconds worth of sample data (i.e. the numbers in the below table are “CPU ms / sec of music output data”, i.e. smaller is better):

wasm

I repeated my measurements multiple times (6x) and eventhough the results were – for the most part – reproducable, they fluctuated considerably (e.g. +/-10%). Any single digit percentage measurement is therefore to be taken with a pinch of salt. In Chrome there were even some massive hiccups (maybe some background garbage collection? see “(*) worst times” in parenthesis). The above table shows the “best” result that I ever observed for the respective scenarios.

Interestingly with regards to the “better performance” claim, the results are not really conclusive yet. There are some finding though:

  • Chrome users may typically experience a massive performance improvement from WASM.
  • FireFox’s asm.js implementation already performs much better that Chrome’s. For Chrome users, WASM here is actually only the 2nd best choice – for most scenarios the performance benefit of switching to FireFox here is even bigger.
  • For FireFox users the situation here is more complicated. It really depends on the specific program: Some may run massively faster, but some may actually run slower than their asm.js equivalent!

PS: I had only briefly looked at Edge but asm.js performance is slightly worse than Chrome’s and WASM is almost 2x slower than Chrome’s.

An important thing that I did not mention yet are startup times: WebAssembly is designed to be parsed more easily than respective JS code, the asynchronous loading may then also speed things up (in case your browser really puts those multiple CPUs to good use..).

And indeed this is where Chrome (and even Edge) actually shines: For the old asm.js version of my page it takes about 3 seconds for Chrome (4 seconds for Edge) to locally load/display it on my PC. For the new WASM version it’s barely more than 1 second (also for Edge)! FireFox somewhat disappoints here: It also improves on the 4 seconds for the old asm.js page, but the new WASM version still takes 2 seconds to load/display (Chrome/WASM may not be too bad after all.).

  • So WebAssembly may not always improve execution speed, but combined with the greatly improved startup time it is really nice!

sprinkler update..

This is just a little update regarding my earlier DIY sprinkler project: The old Rainbird product (see left photo below) had finally died completely and the time had finally come to put in “version 1.0” of my home-grown replacement (see photo on the right).

old     controller

As compared to the original post I had meanwhile replaced the ATmega328P (i.e. the Arduino ProMini) with an ATmega128: This microprocessor gives me more space for the program code – but it still is a very cheap IC.. However it comes at the price of some extra soldering (see green PCB on the photo above). Anyone interested in using this chip as a replacement for less powerful Arduino’s should have a look here: https://github.com/MCUdude/MegaCore

Previously I had been performing all of my testing with brand new 12V DC solenoid valves. The question was whether or not my new 12V DC controller would also work for the old 24V AC valves already in place from the old installation:

valves
To my great relief it works perfectly and I can finally control the things remotely from my PC 🙂

GUI

269 Life

In addition to showing some cool raymarching based realtime graphics, my latest web page is dedicated to “269 Life” and the matching title is meant to attract some extra attention to that meaningful movement (see http://www.269life.com).

The realtime WEBGL web page can be found here:  https://www.wothke.ch/269life/. You’ll need some sort of 3D graphics accellerator and a Chrome browser to use it. Or you can have a look at a youtube recording here: https://www.youtube.com/watch?v=Y3S3MY3Vuf4 (I am still looking for a volunteer to create a higher quality recording for me 🙂 )

I am not repeating the information that can already be found in the comment of the youtube video here. Instead I’ve added some background information regarding the techniques that I used in the page.

This slideshow requires JavaScript.

All the fractal graphics are created using knightly’s “pseudo kleinian” algorithm (see example code in “Fragmentarium”) as a base and parametering it with various “distance estimate” functions. An “orbit trap” based implementation is used to color the result. Depending on the specific “scene” a number of reflections is calculated (up to three). The “phong blinn” shading model is finally used in combination with a standard “ambient occlusion” implementation to render the fractal background (basically the same impls that I had previously used in “modum panem”).

Three different approaches are used to display text elements:

  1. “Flat” texts are created by using a font texture that is then displayed via simple triangles (two per character).
  2. Texts like the title or the ones in the “greetings” section are then based on extruded 3d fonts (see standard THREE.js examples).
  3. Finally there are the “particle based” texts that explode in the “greetings” section – which are created using regular canvas text rendering.

A “bokeh” postprocessing is applied to the resulting page to create a “depth of field” effect. (The respective implementation is derived from Dave Hoskins work.) The “bokeh” postprocessing is also used to create some interesting distortion effects on the overlayed title text (which is not using the same z-buffer).

This slideshow requires JavaScript.

Finally the “greetings” scene showcases the combination of “standard” THREE.js elements (particles, extruded texts, etc) with the shader generated fractal background: By having the fractal shader propagate its z-buffer information, the regular THREE.js overlays are later clipped correctly (thanks to Marius for the respective depth calculation – see boxplorer2).

The “neon signs” here are created via a postprocessing pass that adds a “glow” as well as a “god’s ray” effect. A simple random noise based shader is used to create the purple “northlights” on the horizon and 20’000 confetti particles provide for some action.

Thanks again to Wolf Budgenhagen and Darkmelo for letting me use their music.

modum panem

s1
Obviously there was something missing in my previous mandelbox experiment… exactly, some means of transportation so you can explore the scenery!

s5
I therefore extended the original version such that “regular” 3D stuff can be mixed with the raymarching based fractal landscape.

 
s3

 

 

 

 
also I added a bit of “collision detection” so that the landscape can be moved “out of the way” when your transport gets too close..

 

s2

so enjoy your ride: https://www.wothke.ch/modum/#/wright-and-bastard/sets/evoke-2016

or have a look at the youtube recording: https://www.youtube.com/watch?v=bx0Aakv3JPE

 

 

 

s4

Notice: Unfortunately Chrome seems to be the only browser that currently properly supports WEBGL_draw_buffers. (Firefox also claims to support the feature but in fact it is completely messing up the respective color attachments..

part three..

And to complete the trilogy here another bit of fractal fun (in case your PC is not equipped with a fast enough graphics accelerator you can find a video recording here: https://youtu.be/PzTwOfoZp0E):

screenshot

the beauty of numbers..

I just did a little revival of my old WebGL fractal stuff. The two below screens use the latest version of my Commodore C64 emulator to play two fitting music creations by Markus Klein.

ricemilk

clubhouse

and now to something completely different..

I have some older automated lawn sprinkler installed in my garden. So when the respective controller started to misbehave my first reflex was to try and repair it.. but not having any real clue about electronics, this proved to be somewhat difficult, and with the limited functionality that the commercial product had offered in the first place, I reckoned that it wasn’t worth the effort anyway.

But being tired of performing the same manual irrigation round every one or two days during summer – not to mention the vegetable garden – I thought that it might be a fun little project to design a replacement tailored to my needs, while learning a thing or two about simple electronics.

irrigation project

What I wanted was a system that would not just stupidly irrigate based on some predetermined schedule, but a system flexible enough to adapt based on actual irrigation needs. Also I wanted to be able to collect (on my PC) all weather/irrigation related data so that I will be able to verify that the system actually has the desired effect. It should then be possible to interact with the system without having to rewire the garden. Obviously the system should continue to function (and not lose sensor data) while my PC is switched off.

Side note: As a general principle I am trying not to rely on WiFi at all, but the place where the irrigation is supposed to happen is also actually out of regular WiFi range (unless I were to use additional repeaters). I would have liked to use some powerline based approach but rejected that idee (for now) due to the added costs and engineering complexities. I am not interested in mobile phone based approaches either: First of all the whole point of this exercise is “automation”, i.e. I want to prepare for those moments where I am NOT at home (e.g. when I am on vacation) and where I do not have adequate information regarding which of my plants might be in need of irrigation. Having a phone (or web based) “remote control” for me therefore seems to be but a rather useless gadget. Finally I don’t want to pay for a respective phone contract (that I otherwise do not need) just to irrigate my garden.

This being my first baby-steps in the realm of DIY hardware, I wanted to use something as inexpensive as possible – so that it would not hurt financially regardless of how many electronics components I might fry due to my noobish ignorance.. I therefore decided to base my experiment on some cheap chineese Arduinos. These things cost about a Euro each such that the single most expensive component for all of my devices finally proved to be the plastic case used to package them..
devices

I ended up building three types of devices:
1) soil/air sensors: Equipped with a simple 433MHz transmitter these devices periodically broadcast their measurements, and various power saving techniques make sure that they’ll get through the season on their rechargable 9V battery.

2) RainMaker controller: This controls the irrigation valves (based on configuration and sensor data) and it also buffers and relays (to the PC) the data that comes from the sensors. It uses a somewhat more powerful transceiver for two-way communication with the PC.

3) USB interface: Simple device that is connected to the PC’s USB port. It allows the PC to communicate with the RainMaker device(s) in the field – or any other device that is based on my respective custom communication protocol.

When ordering the parts in small quantities from AliExpress the material costs for the different devices are: 10-12$ per sensor (depending on features), 12$ for the USB interface, 45$ for the RainMaker and another 45$ for the irrigation valves (6-way) assembly.

For the PC side I setup a simple DB to store sensor/irrigation data and I wrote a simple Java GUI used to visualize that data, and to interact with available devices in the field.

workbench_ov

So far I successfully completed the dry-testing, and it will be time to test the devices in the field.. where it remains to be seen what additional insights will come of that.

Lessons learnt so far..

  • With a bit of help from an electronics savy coach (thank you Markus! 🙂 ) doing a bit of DIY hardware can be a fun experience – even for electronics beginners.
  • As soon as a MickeyMouse project gets just a tiny bit “bigger” you may quickly run into the limitations of the inexpensive/smaller Arduino’s hardware (e.g. you want to use a receiver and an additional transceiver, a LCD, a RealtimeClock, some Relais and an EEPROM and you may run out of I/O pins; and even when you circumnavigate that issue (e.g. using I2C and some PISO shift register) you might find that your program – with all the required libs – just barely fits into the available 32k FLASH ROM). Except for very simple devices, optimizing your stuff for size seems to be a part of the Arduino deal. I had last encountered these kind of problems on the C64 some 30 years ago 🙂
  • What people seem to call an IDE in the Arduino world can be rather frustrating for anyone who is used to work with a real IDE (aka IDEA IntelliJ, etc). Calling it “AlmostNotepad” would be a more fitting description. Certainly you don’t want to make programming errors in that kind of environment.. ever 😉
  • Actually there is a friendly / supportive Arduino community that may considerably help you on your learning curve.