Nigel McNie

Nigel McNie http://nigel.mcnie.name/ Website and blog of Nigel McNie, entrepreneur and founder of Shoptime (prior: Mahara and GeSHi). Nigel writes about webapps and being remarkable. (Posts tagged with planetmahara) en Tue, 10 Aug 2010 08:47 NZST Tue, 10 Aug 2010 08:47 NZST nigel@mcnie.name planetmahara Veedub 5 Nigel McNie http://nigel.mcnie.name/static/images/nigel.jpg http://nigel.mcnie.name/ Removing the 'referer' part of PHP Error Messages from Apache Logs http://nigel.mcnie.name/blog/removing-the-referer-part-of-php-error-messagse-from-apache-logs This has always bugged me (scroll right to see the whole thing): <pre class="literal-block"> [Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84] [WAR] 7d (lib/web.php:916) Undefined variable: message, referer: http://mahara-test/admin/extensions/test.php? [Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84] Call stack (most recent first):, referer: http://mahara-test/admin/extensions/test.php? [Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84] * log_message("Undefined variable: message", 8, true, true, "/home/nigel/src/mahara-test/htdocs/lib/web.php", 916) at /home/nigel/src/mahara-test/htdocs/lib/errors.php:446, referer: http://mahara-test/admin/extensions/test.php? [Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84] * error(8, "Undefined variable: message", "/home/nigel/src/mahara-test/htdocs/lib/web.php", 916, array(size 2)) at /home/nigel/src/mahara-test/htdocs/lib/web.php:916, referer: http://mahara-test/admin/extensions/test.php? [Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84] * json_reply(false) at /home/nigel/src/mahara-test/htdocs/admin/extensions/test.json.php:22, referer: http://mahara-test/admin/extensions/test.php? [Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84] , referer: http://mahara-test/admin/extensions/test.php? </pre> I like using log files to display messages instead of the screen, but somewhere along the way somebody decided that the 'referer' (sic) was needed in the logs too, and that makes them totally unreadable. Here's my fix, when tailing a log file: <pre class="literal-block"> nigel@mahara-test:~$ sudo tail -f /var/log/apache2/mahara-test.error.log | perl -pe 's/, referer:.*//' </pre> And, instantly: <pre class="literal-block"> [Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84] [WAR] 7d (lib/web.php:916) Undefined variable: message [Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84] Call stack (most recent first): [Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84] * log_message("Undefined variable: message", 8, true, true, "/home/nigel/src/mahara-test/htdocs/lib/web.php", 916) at /home/nigel/src/mahara-test/htdocs/lib/errors.php:446 [Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84] * error(8, "Undefined variable: message", "/home/nigel/src/mahara-test/htdocs/lib/web.php", 916, array(size 2)) at /home/nigel/src/mahara-test/htdocs/lib/web.php:916 [Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84] * json_reply(false) at /home/nigel/src/mahara-test/htdocs/admin/extensions/test.json.php:22 [Tue Apr 20 01:06:47 2010] [error] [client 192.168.5.84] </pre> Much better! But we can do better again: <pre class="literal-block"> nigel@mahara-test:~$ sudo tail -f /var/log/apache2/mahara-test.error.log | perl -pe 's/\[error\]\s+\[client.*?\]\s+//; s/, referer:.*//' </pre> And we end up with: <pre class="literal-block"> [Tue Apr 20 01:06:47 2010] [WAR] 7d (lib/web.php:916) Undefined variable: message [Tue Apr 20 01:06:47 2010] Call stack (most recent first): [Tue Apr 20 01:06:47 2010] * log_message("Undefined variable: message", 8, true, true, "/home/nigel/src/mahara-test/htdocs/lib/web.php", 916) at /home/nigel/src/mahara-test/htdocs/lib/errors.php:446 [Tue Apr 20 01:06:47 2010] * error(8, "Undefined variable: message", "/home/nigel/src/mahara-test/htdocs/lib/web.php", 916, array(size 2)) at /home/nigel/src/mahara-test/htdocs/lib/web.php:916 [Tue Apr 20 01:06:47 2010] * json_reply(false) at /home/nigel/src/mahara-test/htdocs/admin/extensions/test.json.php:22 [Tue Apr 20 01:06:47 2010] </pre> Hooray! Readable PHP log files. You don't want to have to remember to type all of that, so for best results, put it in a script, <tt class="docutils literal">chmod +x</tt> and you have: <pre class="literal-block"> nigel@mahara-test:~$ sudo tail -f /var/log/apache2/mahara-test.error.log | phptail </pre> You could even combine it with <a href="http://dollyfish.net.nz/projects/monkeytail">Monkeytail</a> to make viewing the logs even easier ;). I've trawled through the PHP codebase looking for where it adds the referer in a vain attempt to change it, or provide a patch to make it a configuration option. Sadly, I never found it. If anyone finds out, please let me know! Thanks to <a href="http://dollyfish.net.nz/">Martyn</a> for the original idea, and assisting with the arcane perl stuff. What's up in Mahara land recently? http://nigel.mcnie.name/blog/whats-up-in-mahara-land-recently Here's a snapshot from my point of view about what's going on across all aspects of Mahara. <div class="section" id="hackfests"> <h1>Hackfests</h1> Yesterday, <a class="reference external" href="http://she.geek.nz">Penny</a>, <a class="reference external" href="http://seld.be">Jordi</a> (both from Mahara partner <a class="reference external" href="http://liip.ch">Liip</a>) and I had another weekend hackfest. Penny has been adding a framework for unit testing, and managed to get the first test working. Jordi was working on swapping <a class="reference external" href="http://smarty.net">Smarty</a> for <a class="reference external" href="http://dwoo.org">Dwoo</a>, while I continued hacking on <a class="reference external" href="http://mahara.org/interaction/forum/topic.php?id=917">View Themes</a> support. Pics: <a class="reference external" href="http://temp.mjollnir.org/firstswissmaharahackfest.jpg">Switzerland end</a>, <a class="reference external" href="http://stuff.nigel.mcnie.name/nz-hackfest.jpg">NZ end</a>. Penny has been working on unit testing, as it's something we developers have realised has been lacking in Mahara for some time now. What unit tests do is give us more confidence that when we make a change (whether it's a bug fix or new feature), that the rest of Mahara continues working exactly the way it used to - i.e., we didn't inadvertently add any more bugs. This has the result that we can continue development safe in the knowledge that we didn't break anything that worked before, which will result in better quality. It will also make it easier for enthusiastic hackers to make changes to Mahara and know they haven't broken anything important. Unit testing isn't the only way we've been trying to integrate testing - Gerald Quimpo from <a class="reference external" href="http://catalyst.net.nz">Catalyst</a> has been resurrecting the <a class="reference external" href="http://seleniumhq.org/">Selenium</a> tests that come with Mahara, and has done an excellent job in getting them working again, while also setting up a Continuous Integration server (<a class="reference external" href="https://hudson.dev.java.net">Hudson</a>) so the tests will be run regularly. Soon, we'll have that available publicly so everyone can see which tests are currently passing and failing. Backtracking a tiny bit - Jordi has been working on swapping Smarty for Dwoo, a change we would like to do for a couple of reasons. Firstly, Smarty is bigger and slower, perhaps largely due to the fact that it needs to support PHP4 (which Mahara doesn't support). Secondly, Smarty doesn't make it easy for us to output variables that are HTML escaped by default - which is currently the leading cause of security vulnerabilities in Mahara. We haven't had many security vulns actually, Mahara is reaping the benefits of a security-oriented development ethic. But of the few we've had, almost all of them have been because someone forgot to put <tt class="docutils literal">|escape</tt> on the end of a smarty variable. So switching to Dwoo will cut down that entire avenue of attack. </div> <div class="section" id="mahara-1-2"> <h1>Mahara 1.2</h1> We <a class="reference external" href="http://mahara.org/interaction/forum/topic.php?id=963">released the first beta of Mahara 1.2</a> last week. That means we're looking for bug reports now, and encourage people to download it and try out the new features. Here's a short list of the coolest new features in 1.2: <ul class="simple"> <li><a class="reference external" href="http://wiki.mahara.org/Developer_Area/Import//Export">Import and Export</a> - Mahara supports the LEAP2A standard for import/export, and in addition, allows you to export an HTML version of your portfolio (including a nice HTML represntation of an individual View).</li> <li>"My Files" and file handling improvements - now the My Files section makes it easier to quickly upload files, and degrades to work without javascript. You can now upload files directly into Views, and use a standard directory browser to choose files in Views. For developers, we created a pieform element for uploading/attaching files, which will make it easier than ever to do this in custom plugins.</li> <li>Theming - we're shipping with 6 themes, to give people an idea of how they can customise their Mahara. We've done a lot of rework of how themes are created, so from 1.2 it'll be even easier to create a custom theme. We even documented the process (finally!).</li> <li>Usability/Comprehension tweaks - Now groups get a default forum, and blogs get a default blog. If there is only one forum or blog, users are taken straight to it. This should really help in preventing the problem where people create a blog as if it's a blog post, and should make the forums an even more useful tool.</li> <li>Improved Moodle/Mahara integration - with a custom patch to Moodle 1.9, you'll be able to submit a View to Moodle for assessment as an assignment answer. How useful is that? Now people can keep their data in their portfolio, and mash it together into a View for assessment over on Moodle, where all the useful grading functionality is :)</li> </ul> There's s still a bit of work to go before Mahara 1.2 will be ready though. I need to finish off LEAP2A import - currently resume import is only half finished, and view import is just begun. We'll also need a fancy UI for it, and of course there are some inevitible bugs that will need squashing. Also, I'd really like to get view themes to a point where it can be merged also. That'll really help people personalise their environment. If I get around to it, maybe we can add a theme chooser for users to change how they see the entire site, not just individual Views. Other than that though, I think we're looking pretty good! </div> <div class="section" id="code-contribution-policy"> <h1>Code Contribution Policy</h1> We are currently preparing a policy for contributors to the codebase. We aren't going to be wanting copyright assignment, but we are going to make sure all contributions are as legal as we can make them ;). Unlike many open source projects, Mahara is actually in a pretty good position legally, as most members of the team are well aware of the legal issues around open source and licensing, and we take legal issues seriously. Our policy won't be onerous, but will ensure we stay legit now and into the future. </div> <div class="section" id="mahara-1-3"> <h1>Mahara 1.3</h1> Though 1.2 is not out yet, talk is already shifting in some departments to the plans we have for Mahara 1.3. The big focus there will be usability, and we're going to target that in a couple of main areas - the navigation, and View creation. For the navigation, our current plan is to adopt a dropdown menu structure, organised more to how users think when they're looking for something. The main categories will be roughly along the lines of "My Stuff", "Friends & Groups" and "Sharing". We'll build a demo of this at some point after 1.2 so the community can see it in action and give feedback, though I've seen it in action elsewhere and it's easy to grasp. Furthermore, the addition of "horizontal navigation" - where when you're at one page you're given suggestions about what else you might be looking for - should help tremendously as well. We're also going to have a good look at the View creation process. The Drag and Drop and layout concepts won't be changing, but the layout of the editing screens themselves will be improved a lot. We're hoping we can even ditch a step to make things simpler. I think the biggest gains will be seen on the View Access screen, which currently can be a bit of a minefield to navigate. We'll get mockups out some time after 1.2 also. Other stuff we might do could include Group Views, a User Dashboard (as a View of course!), and <a class="reference external" href="http://mahara.org/interaction/forum/topic.php?id=956">overhauling the notification system</a>. </div> <div class="section" id="mahara-growth"> <h1>Mahara Growth</h1> And last but not least, here are some interesting statistics about Mahara and its community: We now have over 3000 members of the <a class="reference external" href="http://mahara.org/group/view.php?id=1">mahara.org community</a>, with <a class="reference external" href="http://mahara.org/partners">17 partners worldwide</a>. The community growth has been steady since we launched the new mahara.org last November. The site receives nearly 25,000 visits a month (for the geeks among you: 57% firefox and 27% IE). Of these visitors, we get the most from the US, followed by the UK, Germany, Spain, Australia and then New Zealand. Since we added the ability for sites to register on mahara.org in May, we now have over 50 sites doing this every week. These sites together add to some 12,000 users, but this doesn't include mahara.org, or the New Zealand <a class="reference external" href="http://myportfolio.ac.nz">MyPortfolio</a> <a class="reference external" href="http://myportfolio.school.nz">service</a> - if these numbers were included, the total would rise to nearly 25,000. Again for the geeks - we are seeing a 75/25 split of MySQL to PostgreSQL. Hopefully things will skew more to the PostgreSQL side of that equation as time goes on :). Of the registered sites, the largest seem to be run by Blackpool University, Uni Krems (Austria) and Birmingham City University, all of which have thousands of users. </div> Mahara 1.2 Import/Export Demo http://nigel.mcnie.name/blog/mahara-12-importexport-demo OK so my last post was just all text, which is a slog to read through. So here, I made a video about Mahara 1.2's import/export: <div class="center"><object width="425" height="344"><param name="movie" value="http://www.youtube.com/v/tnFx0NHn4Ls&hl=en&fs=1"></param><param name="allowFullScreen" value="true"></param><param name="allowscriptaccess" value="always"></param><embed src="http://www.youtube.com/v/tnFx0NHn4Ls&hl=en&fs=1" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="425" height="344"></embed></object></div>I don't think I got the sound levels right and I'd have loved to have added a background track, but video editing on linux is a pain. It's only my second video so be nice. The next one should be better again ;) Themes In Mahara 1.2 http://nigel.mcnie.name/blog/themes-in-mahara-12 One of the things we're doing for the upcoming 1.2 release of Mahara is doing a big tidyup of the theming. There are some code level changes, but the most important changes are to do with the themes themselves. Note that I said "themes". This is because 1.2 won't have just one theme - it'll have six. One is a brushed up version of the default, one is "special" (more on that in a minute), and the other four are all-new themes that will give Mahara users a chance to immediately personalise their installation "out of the box". The new themes are mainly just colour changes, but they're pretty damn good looking colour changes all the same. They'll also give people good examples to work from when creating their own themes. Now, the special theme. Mahara will now have a theme called 'raw', which is a very stripped down theme. There's very few colours or styles, and most of the theme's job is to position everything correctly and set sensible defaults. All other themes extend from this theme. Mahara has a theme inheritance system, where by you can specify another theme as the 'parent' of your theme, and thus rely on it to provide the templates and most of the CSS. From now, every theme will ultimately extend from the 'raw' theme - including the default theme. The inheritance can have any number of levels, so for example a theme someone did for Mahara 1.1 might have a parent of the default theme which might be parented by the raw theme. Sadly, because we haven't documented the theme inheritance well, some of the themes out there have copied the entire default/ directory to create their new theme. These people will probably have noticed that when new versions of Mahara come out, sometimes bugs they are supposed to fix are not fixed - this is because the fix was made in a 'default' theme template, which is overridden by their theme's copy. When they upgrade to 1.2, it's highly likely their theme will totally break. Elsewhere on the backward compatibility front, it should be easier though. If your theme uses theme inheritance, then it will continue to inherit from the (slightly changed) default theme. You may also have to change some CSS selectors in your stylesheet to match the new HTML structure. Speaking of which, we've cleaned up the HTML immensely, and the CSS along with it. I think the default stylesheet was over 50KB in previous versions of Mahara, at the moment I have the raw theme down to much less than half that, and I don't think it will end up much bigger than 25KB. The default theme's stylesheet will be negligible in size. We're also going to cut the print stylesheet down to size, and if we get the chance we'll add in RTL and browser-specific stylesheets too. The net result of all of this is a bit of short-term pain while people migrate their themes, but a lot of long-term gain, as the HTML will be cleaner, the CSS smaller and every Mahara faster because of it. Theme development will be much easier from 1.2 onwards, I'll make sure we write documentation so people don't get caught out by the theme inheritance issue again. Maybe at some point after 1.2 we can look at some fine-tuning and performance of the theming system. I'd love to crank out some CSS sprites and javascript minification, but right now I don't think there's going to be much time other than for 'easy wins', such as expires headers and configuring mod_deflate (which Mahara does already). Anyway, that's enough from me for now. We have this theming work and finishing off import/export to do, followed by a lot of bug fixing before 1.2 can be released. Look out for another alpha soon, at least. Mahara 1.1 - almost there! http://nigel.mcnie.name/blog/mahara-11-almost-there There's three bugs and two feature requests left until 1.1 is ready to roll! Finally it looks like we've got on top of the long list of issue and will be able to get a release out later this month. Here's a quite explanation of how things will progress over the next while: Once the bugs and feature requests are fixed, we will release the first, and possibly only, release candidate of 1.1. This will be a release that we consider "good enough" for 1.1 - no major crashing bugs, upgrade from 1.0 works, new functionality works. We'll let that circulate for a few days, to give people a change to have one last go at testing upgrades and looking for major bugs. If there's nothing reported that's too major, we'll do the 1.1 release, otherwise, there will be a second (and potentially third) release candidate. The 1.1 release will no doubt need some tweaking and minor fixes, which I'm sure will be reported back to us quickly. So there will probably be a series of point releases after 1.1 to iron out the kinks. We'll fully support you if you upgraded from any version of 1.0 to 1.1 though, and hopefully the testing we've done means the upgrade path doesn't break, even if some minor things don't work in 1.1. Meanwhile, we'll be working hard on the import/export functionality for 1.2. This is the only feature planned for 1.2, and we hope to have it released quite quickly, by mid this year even. So, look out for Mahara 1.1 RC1 real soon now! Things You Don't Want to See in the Morning http://nigel.mcnie.name/blog/things-you-dont-want-to-see-in-the-morning <pre class="literal-block"> maharademo=# select count(*) from activity_queue; count ------- 10043 (1 row) </pre> Aah crap. So it turns out there's a bug in Mahara's activity queue processing which means that if you make a view, leave objectionable feedback on it then delete the view before the queue is processed, the queue will bomb out every time it gets to processing the feedback and not send any of the other messages. Doh! I have written a patch for this that'll go to 1.0_STABLE and master. It will probably involve unconditionally deleting any notifications older than two weeks, as the last thing people want is old notifications. This problem may have affected other Mahara installs out there, but it's not guaranteed to have. For example, it hasn't affected the <a class="reference external" href="http://myportfolio.school.nz/">MyPortfolio</a> <a class="reference external" href="http://myportfolio.ac.nz">sites</a>, nor <a class="reference external" href="http://mahara.org/">mahara.org</a> itself. You can find out by doing a <tt class="docutils literal">SELECT COUNT(*) FROM activity_queue;</tt> on your database - large numbers are bad, 0 or small numbers are good. There will probably be a 1.0.7 soon to fix this. mahara.org finally relaunched! http://nigel.mcnie.name/blog/maharaorg-finally-relaunched It's taken a while, but finally the <a class="reference external" href="http://mahara.org/">new mahara.org</a> has been launched. I think it's a great step up from the old site, and will hopefully serve as a useful place to showcase and document Mahara for a while yet. DNS is still propogating so it might be a while until everyone can see the new site. In the mean time, we're putting up some content and will tweak the site over the next couple of days. Our focus will now turn back to the Mahara 1.1 release. There's still some bugs to iron out, but hopefully we can get the release out soon...