SysAdmin'ish Blog
Munin 2.0: "Date::Manip::Backend" error
Munin as in Debian, currently version 2.0.6, features a noisy error, at least in my servers' Apache error.log:
munin-cgi-graph: Name "Date::Manip::Backend" used only once: possible typo at /usr/lib/munin/cgi/munin-cgi-graph line 31.
This is due to a bug. It should have been dealt with already (DBTS #561970) but either not sufficiently or something else is going on.
A reasonable workaround is to add no warnings; before line 31 in /usr/lib/munin/cgi/munin-cgi-graph:
BEGIN { no warnings; $Date::Manip::Backend = 'DM5'; }
Yet Another or fix DokuWiki blogging?
What requires less time? Writing a new blog generator or fixing your old? — I can tell you my answers were changing rather often within the last 12 hours 
Eventually, I ended up fixing my blogs (including this one here) running on DokuWiki and the blog plugin.
Since many years, I am happily using DokuWiki for documentation and to publish scripts and such. In 2007, I started to also use the blog plugin for blogging about technical stuff. I love the simplicity, and the plain text flatfile backend. However, the experience has always been rather troublesome. Plugin updates often broke the layout, had vulnerabilities, needed fixing, or just didn't work.
Naturally, I was constantly looking for alternatives (why abuse a wiki for blogging anyhow?). I was hoping to find a simple, static blog generator that I could run from the command line. Of course, there are plenty. Strangely enough, though, when looking closer only very few withstood my criteria. As time passed by I started to sketch out some ideas for writing my own blog generator: Yet Another Shell Blog Engine (YASHBE) sounded like a cute name 
Eventually, I created a comparison table. At first, doing so seemed to be a very good idea because my first thoughts were "Oh, look, the DokuWiki thing is not that bad after all" and "Well, my own code could do a lot if I wrote it." – To cut a long story short, reviewing alternatives, writing the comparison table, researching and fixing the problems with DokuWiki took more than 12 hours. If I had started to code YASHBE right away I guess I would have finished it by now, too.
Problems with DokuWiki
- The blog index page was partly broken and looked horrible which is why I updated it manually for some time. Fortunately, thanks to work by Michael Hamann that was fixed in the "blog" plugin.
{{blog>blog?7&nouser&permalink&noeditbutton}}provides reasonable results for me. - RSS feeds could only be called broken.
Apparently, older versions of plugin:feed could be used for blog posts but I could'nt get this work with the current version.1) The documentation recommends to use DokuWiki's native RSS syndication, however, results were unacceptable. - Archive pages looked horrible, workarounds using plugin:pagelist didn't fit my bill.
Fixing RSS feeds
After reading the source code of feed.php I found that actually there is a way to exclude "media" (file uploads). It just wasn't documented. So,
[[this>feed.php?ns=blog&linkto=current&content=html&mode=recent&view=pages&num=7|Blog posts]]
was pretty close to what I expect from a blog feed: Pages only (no uploads), linking to the current version. Due to a bug num=7 is ignored, but the feed delivers at least 5 entries which should be sufficient.
Update of 2013-04-21:
feed.php delivered less than 7 entries because the number of matching entries in meta/_dokuwiki.changes was lower than 7. See my comment on DW bug #2765.
Unfortunately, the titles of the feed items include the "edit summaries". So, if I fix a typo in a post entitled "Feed woes", and if I document this in the edit summary, the feed item will be e.g. "Feed woes - typo fixed". This is hard-coded in feed.php, line 227 in the "Adora Belle" version. I've added && ($_GET['ns']!='blog') at
if($conf['rss_show_summary'] && ($_GET['ns']!='blog') && !empty($ditem['sum'])) { $item->title .= ' - '.strip_tags($ditem['sum']); }
This or a similar switch should do.
Update of 2013-04-21:
Today, I found that the "blog" component of plugin:feed works quite OK and I doubt that it is going to be removed (as it says on its web page). So, one can use e.g. {{blogfeed>blog?7|Blog posts}} to get a feed of 7 items without the need to hack feed.php. However, the {{blogfeed...}} shows only abstracts, not full text entries.
Fixing plugin:feed
Update of 2013-04-21:
The feeds produced by plugin:feed have a hard-coded favicon.ico pointing at lib/images/favicon.ico. Consider to add the file, hack plugins/feed/feed.php or add e.g.
RewriteRule ^lib/images/favicon.ico /favicon.ico [L]
to .htaccess.
Fixing the look of archives
For SysAdmin'ish Blog Archive I use {{archive>blog?2013-*&nouser}} etc.. This generates an HTML table for each year. Since these are separate tables the column widths vary. It's just cosmetics but IMHO it really hurts. I could use {{archive>blog?*&nouser}} to generate 1 large table for all entries but I wanted the year headlines. The following CSS improves the look (on my template):
/* make tables full width */ table.pagelist { width: 100%; text-align: left; } /* reserve some fixed width for the page titles */ table.pagelist td.page { width: 47%; } /* make date and comment columns non-wrapping and non-expanding */ table.pagelist td.date, table.pagelist td.comments { white-space: nowrap; width: 1px; }
Conclusion
Added 2013-04-21
I wanted to add this as a reminder (to myself and whoever cares): I am normally much in favor of recycling and fixing the old instead of getting new stuff or re-inventing things. However, in this case it seems like if I had started to write my own program right away, without checking out alternatives, without making a list of what I actually disliked about the old solution, without researching bugs and whether they might be easy to fix, by now, I would have a working solution, too, plus a lot of hacking fun and the joy of learning new stuff (my plan was to use Markdown and Bootstrap).
668.5
Of course, you know that 42 is The Answer to the Ultimate Question of Life, The Universe, and Everything.
So, clearly, you know that 21 is only half the truth2).
However, do you also know what 668.5 is?
No? — Well, it's the average 
How to kill ethernet with USB
Have you ever told a friend or client that s/he could wire up the PC and external devices all alone because plugs only fit into 1 specific socket and cannot be mixed up? And if there is more than 1 socket of the same type they are color coded (like for PS/2 and audio sockets)? — I did, but I won't do it again, probably, at least not if I have to assume that a standard USB wire of type A-B is around!
I thought I had seen it all, like one guy who plugged in a 15-pin VGA wire upside-down. In former times, every sysadmin knew to warn people that phone wires with smaller RJ plugs do fit perfectly into RJ45, and in fact can cause great damage. Fortunately, these wires are pretty much extinct nowadays. I really thought it was safe to let users do the wiring.
Today, a friend of mine called me because he couldn't install his new printer. The printer offered LAN and USB. My friend made sure the printer works by itself but when connected to a PC via USB the PC did not find any printer.
Well, of course not! I had no idea how damn well a Type B USB plug fits into a RJ45. Besides, I don't want to know how much damage it can cause. It short-circuits at least 6 pins! However, for my friends sake I'll now go and check it out.
No new devices could be found
We got a computer with rather new hardware and no operating system pre-installed. When I wanted to install Windows 7 it prompted me for drivers which I provided, but no matter what, Windows 7 refused to continue the installation and kept telling me
No new devices could be found
The computer was equipped with an Intel DH61AG board and an SSD drive. In the BIOS we activated AHCI. So, I was not surprised that Windows asked for drivers.
First, I thought I got the wrong drivers. I double-checked and I tried different ones from the CD that came with the computer. No go. Second, I downloaded up-to-date drivers for the "F6 floppy" (apparently some manufacturers still list them in this category), but this did not help either.
At that point I started to [guugl] for the error. Sure enough there were plenty of postings, however, none of the suggested solutions seemed appropriate. I had the right drivers for the right architectures. The hard disk was OK (I could boot Ubuntu 12.04 without any problems), and so on. Anyway, just for the record, one more thing I tried was completely erasing the hard disk, though to no avail.
One poster said that the problem was a supposedly faulty USB stick which was used for the installation. This got me thinking because I was using a (virtual) CD-Rom connected via USB. So, I tried a (real) Windows 7 DVD instead. This, indeed, worked without any problem. In fact, Windows 7 didn't even ask for drivers. It showed me the license agreement, asked for the installation mode and then offered me a list of partitions from the hard disk.
I stopped the installation right away, because I wanted to know what was going on. A few more experiments, and I knew what really was the problem:
I had my installation disk (the Windows 7 DVD) plugged into a USB 3 connector! I guess, every time Windows tried to load the drivers for the AHCI and/or Intel (Rapid) Storage the drive on USB 3 got disconnected. The error message was perhaps misleading.
Once I plugged my installation disk into the normal USB socket everything worked as expected. No drivers needed.
Courier IMAP & POP3 stats per user
So far, I have been using courier-analog from the Courier Mail Server to generate IMAP bandwidth reports on a per user basis. However, recently, when I copied a user's mailbox of about 600 MB from a remote server via imapsync I was surprised that courier-analog showed only a few KB of traffic. First, I thought that courier-analog only counts outbound traffic, but looking at the source code it seems that it counts only the body values (see below). So, I quickly wrote my own script to get per user stats.
A typical IMAP log line on my server running courier-imap version 4.8.0-3 (from Debian) reads
Jul 29 06:36:47 server imapd-ssl: LOGOUT, user=example, ip=[::ffff:127.0.0.1], \ headers=0, body=0, rcvd=48, sent=292, time=344, starttls=1
Courier IMAP distinguishes headers, bodies, received and sent bytes. Clearly, headers and body can be zero while there is still some traffic. But more importantly, body gives only the bytes of messages sent, apparently:
Jul 23 22:12:28 server imapd-ssl: DISCONNECTED, user=example, ip=[::ffff:127.0.0.1], \ headers=0, body=0, rcvd=14084220, sent=8141, time=123, starttls=1
For simplicity I opted to preprocess the logfile with sed and then sum up the received and sent values with awk.
Script
- courier-traffic-imap
#!/bin/bash -e # # Calculate traffic of Courier IMAPd per user [[ $1 ]] && DATE="$1" || DATE=$(date +%b\ %e -d yesterday) [[ $1 == today ]] && DATE=$(date +%b\ %e) LOGFILE=${2:-/var/log/mail.log} THRESHOLD=2900000 # (almost 3 MB) UNIT=2 # Bytes (B): 0, KB: 1, MB: 2, GB: 3, ... echo -e "== Courier IMAP traffic stats per user of $DATE ==\n" # We preprocess $LOGFILE with sed, extracting only imapd lines, # the user name, received and sent bytes. # Numbers are then summed up by awk, and finally pretty printed # We use uniq to work around a bug in courier imap that fills the logs # with thousands of identical lines sed -nre " s/^$DATE (..:..:..) .* imapd.* user=([^,]*),.* rcvd=([0-9]+), sent=([0-9]+).*$/\\2 \\3 \\4 \\1/p " $LOGFILE \ | uniq \ | awk -v threshold=$THRESHOLD -v unit=$UNIT ' # sum up numbers: uc=count, ui=received, uo=sent, u(i/o)g=global totals { uc[$1]++; ui[$1]+=$2; uo[$1]+=$3; ucg++; uig+=$2; uog+=$3; } END { # get unit string (ustr) and exponent (uexp) split("B KB MB GB TB PB",units); ustr=units[unit+1]; uexp=3*unit # print results for (user in uc) { u[user] = ui[user]+uo[user] # per user sum of received and sent if (u[user] > threshold) printf "%28s %6d %-2s [#%5d: r %5d s %5d ]\n", user,u[user]/10^uexp,ustr,uc[user],ui[user]/10^uexp,uo[user]/10^uexp \ | "sort -nr -k 2" } close("sort -nr -k 2") printf "\n%28s %6d %-2s [#%5d: r %5d s %5d ]\n", "-- total --",(uig+uog)/10^uexp,ustr,ucg,uig/10^uexp,uog/10^uexp } '
Example
$ bin/courier-traffic-imap 'Jul 23' /var/log/mail.log.1
== Courier IMAP traffic stats per user of Jul 23 ==
example 624 MB [# 3: r 0 s 623 ]
another-user 90 MB [# 36: r 40 s 50 ]
and.another-user 89 MB [# 7: r 85 s 4 ]
and.so.on 68 MB [# 92: r 10 s 58 ]
and-so-forth 42 MB [# 20: r 31 s 10 ]
-- total -- 1002 MB [# 1027: r 207 s 795 ]
POP3 stats
The logfile format is exactly the same. Just replace every imapd with pop3d. Of course, it would be easy to put both into 1 script. However, POP3 traffic is much lower than IMAP traffic. I guess I'll ignore it completely.
RSS feeds: Blog posts (abstracts), Blog posts (full text), Comments
See also: SysAdmin'ish Blog Archive & Tag cloud
disclaimer & imprint :: copyright :: backlinks :: page history :: index :: recent changes (all) :: go to top ::