Meticulously Planned

I finally got fed up with the RSS readers out there for GNOME. I tried hacking on them to add the features I wanted, and fix the UI. Unfortunately, much of the codebase out there was crackers. I decided to take advantage of python-feedparser, python-gtk2, python-notify and such to quickly hack up a RSS viewer I can stand.

Not really functional yet, but its getting there rather quickly.



http://hg.vwdude.com/marina

I had been contemplating writing a bubble text GtkCellRenderer for a while. Originally, the idea was to write my own RSS reader and use it to list the number of items in a RSS channel. Turns out someone else wanted to use this for something, so I went ahead and prototyped it.

• fancy-cell-renderer-0.1.tar.gz


• fancy-cell-renderer-0.2.tar.gz - Includes fix for renderering properly on HighContrast theme

I started work on the dependency resolution for moneyclip. It is a bit hairier when working in C compared to something like C#. You need to take things into account like ABI matching since this is not done for you by a runtime engine such as in C#.

Important stuff is here.

So I've been working on a hidden C port of Moneyclip. Today I found a great article on implementing plugin architectures with glib from the guys at Imendio. It happened to be from the 2006 GUADEC.

While its not really usable yet, the plugin architecture has landed here.

Now that this is in, I should be able to start on the user interface. I still need to port the command system, qif parser, and top-level data structure for files. In the port, I've implemented support for double-entry accounting.

So I've been toying with building a simple log searching web-app. There are plenty of services to monitor and when something goes wrong finding a solution must be speedy.

There are a few tutorials out there to get you rolling on logging syslog-ng to PostgreSQL. However, they are quite basic and are pretty much limited to a single table. Since I would like to have relationships between at minimum logs and the host that sent them, I wrote a little bit of code to make this possible.

So lets start by getting our database setup correctly. I'm assuming you know how to create users, databases, and secure them.

CREATE TABLE logs_host (
id integer NOT NULL,
name character varying(200) NOT NULL,
label character varying(200),
created_at timestamp with time zone NOT NULL,
modified_at timestamp with time zone NOT NULL
);

CREATE SEQUENCE logs_host_id_seq
INCREMENT BY 1
NO MAXVALUE
NO MINVALUE
CACHE 1;

CREATE TABLE logs_host (
id integer NOT NULL,
name character varying(200) NOT NULL,
label character varying(200),
created_at timestamp with time zone NOT NULL,
modified_at timestamp with time zone NOT NULL
);

CREATE SEQUENCE logs_log_id_seq
INCREMENT BY 1
NO MAXVALUE
NO MINVALUE
CACHE 1;

Now lets create a PLPGSQL based function for PostgreSQL which will handle building our foreign-key relationships and inserting data into the database.

CREATE OR REPLACE FUNCTION syslog(varchar, varchar, varchar, varchar, varchar, varchar, varchar, varchar, varchar, varchar, varchar, varchar, text) RETURNS boolean AS
'
DECLARE

_HOST ALIAS FOR $1;
_FACILITY ALIAS FOR $2;
_PRIORITY ALIAS FOR $3;
_LEVEL ALIAS FOR $4;
_TAG ALIAS FOR $5;
_YEAR ALIAS FOR $6;
_MONTH ALIAS FOR $7;
_DAY ALIAS FOR $8;
_HOUR ALIAS FOR $9;
_MIN ALIAS FOR $10;
_SEC ALIAS FOR $11;
_PROGRAM ALIAS FOR $12;
_MSG ALIAS FOR $13;
hostid integer;

BEGIN

SELECT INTO hostid id FROM logs_host WHERE name = _HOST;

IF NOT FOUND THEN
SELECT INTO hostid nextval(''logs_host_id_seq'');
INSERT INTO logs_host (id, name, created_at, modified_at) VALUES (hostid, _HOST, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP);
END IF;

IF hostid IS NULL THEN
RAISE EXCEPTION ''There was an error locating or creating a host entry for (%)'', _HOST;
END IF;

INSERT INTO logs_log (host_id, received_at, sent_at, facility, priority, "level", tag, program, text)
VALUES (hostid, CURRENT_TIMESTAMP,
(_YEAR || ''-'' || _MONTH || ''-'' || _DAY || '' '' || _HOUR || '':'' || _MIN || '':'' || _SEC || '' -0:00'')::timestamptz,
_FACILITY, _PRIORITY, _LEVEL, _TAG, _PROGRAM, _MSG);

RETURN ''t'';

END;
' LANGUAGE 'plpgsql';

You may need to perform

CREATE LANGUAGE 'plpgsql';

inside your database beforehand so the function gets created correctly.

You might notice that the database tables are aptly named to be used with Django ;-)


Next lets configure our syslog-ng. I'm assuming you are running Ubuntu where the default all-inclusive 'source' is named s_all. Add the following to /etc/syslog-ng/syslog-ng.conf.

destination d_postgres {
pipe("/tmp/pgsql.pipe"
template("SELECT syslog('$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR', '$MONTH', '$DAY', '$HOUR', '$MIN', '$SEC', '$PROGRAM', '$MSG');\n")
template-escape(yes)
);
};

# log to postgresql
log {
source(s_all);
destination(d_postgres);
};

Last, lets create a temporary fifo pipe to PostgreSQL. You would want to secure this in a production environment.

mkfifo /tmp/pgsql.pipe
psql -U postgres -h localhost syslog < /tmp/pgsql.pipe

Now, restart syslog-ng!

/etc/init.d/syslog-ng restart

I had the need to parse some logs for data-archival and whatnot, so I thought I would share it with those who come across this.

/^([a-zA-Z]+)\s+(\d+)\s(\d+):(\d+):(\d+)\s
([a-zA-Z0-9\.]+)\s([a-zA-Z0-9\.-\[\]]+):(.*)$/

This will not match to all syslog entries of course, since everyone likes to store in their favorite format. However, it does match what I'm working with, so I have provided a sample below.

Mar 5 11:11:21 10.0.0.1 dhcpd: DHCPDISCOVER from 00:55:13:89:11:3a via eth0: network 10.0.0/24: no free leases

Happy log parsing!

After a rough 6 months using typo I have decided I cannot stand to use it any more. Do all Ruby-on-Rails applications feel as brittle as typo did?

My completely uneducated guess would include the following as reasons for me not being satisfied.

• Mongrel
• ProxyPass to Mogrel from Apache2
• SQLite
• Ruby and Rails are perhaps just too slow

I just cannot be sure, and will not waste my time finding out.

Labels: Blogging, Ruby