<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Felipe Alfaro Solana &#187; MySQL</title>
	<atom:link href="http://www.felipe-alfaro.org/blog/category/mysql/feed/" rel="self" type="application/rss+xml" />
	<link>http://www.felipe-alfaro.org/blog</link>
	<description>A little bit of technology, security and networking with Linux, FreeBSD and Mac OS X, plus some personal opinions.</description>
	<lastBuildDate>Sun, 23 Oct 2011 16:46:32 +0000</lastBuildDate>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
	<generator>http://wordpress.org/?v=3.3</generator>
		<item>
		<title>Configuring syslog-ng to store logs into a MySQL database</title>
		<link>http://www.felipe-alfaro.org/blog/2005/12/21/configuring-syslog-ng-to-store-logs-into-a-mysql-database/</link>
		<comments>http://www.felipe-alfaro.org/blog/2005/12/21/configuring-syslog-ng-to-store-logs-into-a-mysql-database/#comments</comments>
		<pubDate>Wed, 21 Dec 2005 12:59:14 +0000</pubDate>
		<dc:creator>Felipe Alfaro Solana</dc:creator>
				<category><![CDATA[MySQL]]></category>
		<category><![CDATA[Security]]></category>
		<category><![CDATA[Syslog]]></category>

		<guid isPermaLink="false">http://felipe-alfaro.org/blog/?p=86</guid>
		<description><![CDATA[This short article describes how to configure syslog-ng in order to store the logs into a MySQL backend. This adds more flexibility when performing log analysis, log searching and correlation. Installing MySQL MySQL can be compiled from source and installed using the FreeBSD ports collection: # cd /usr/ports/databases/mysql41-server/ # make install distclean Configuring MySQL Use [...]]]></description>
			<content:encoded><![CDATA[<p>This short article describes how to configure <em>syslog-ng</em> in order to store the logs into a MySQL backend. This adds more flexibility when performing log analysis, log searching and correlation.</p>
<h3>Installing MySQL</h3>
<p>MySQL can be compiled from source and installed using the FreeBSD ports collection:</p>
<pre>
<div># cd /usr/ports/databases/mysql41-server/
# make install distclean</div>
</pre>
<h3>Configuring MySQL</h3>
<p>Use the following commands to create the MySQL database directory and install a configuration file:</p>
<pre>
<div># mkdir /var/db/mysql
# cp /usr/local/share/mysql/my-small.cnf /var/db/mysql/my.cnf</div>
</pre>
<p>Optionally, edit <code>/var/db/mysql/my.cnf</code> to adjust some parameters like:</p>
<ul>
<li>Disabling networking support.
<p>This can be done by uncommenting the <code>skip-networking</code> option from the configuration file.</p>
<p>This will prevent MySQL from listening on port 3306/tcp. Since we are using MySQL locally, we can use UNIX sockets instead of true networking.</li>
<li>Reducing memory usage.
<p>By adjusting <code>innodb_buffer_pool_size</code> and <code>innodb_additional_mem_pool_size</code> to values suited to the amount of RAM available to FreeBSD</li>
</ul>
<p>Next, add the following lines into <code>/etc/rc.conf</code> so that MySQL will get launched during system startup:</p>
<pre>
<div># MySQL
mysql_enable="YES"
mysql_limits="YES"
mysql_dbdir="/var/db/mysql"
mysql_args=""</div>
</pre>
<h3>Starting MySQL</h3>
<p>Use the following command to start MySQL:</p>
<pre># /usr/local/etc/rc.d/mysql-server.sh start</pre>
<h3>Creating the MySQL database</h3>
<p>The logs will get stored into a table named <b>logs</b> on database <b>syslog</b>.<br />
To create the database and table, create a file named <code>syslog.sql</code> with the following SQL commands:</p>
<pre>
<div>CREATE DATABASE syslog;
USE syslog;
CREATE TABLE logs (
	host varchar(32) default NULL,
	facility varchar(10) default NULL,
	priority varchar(10) default NULL,
	level varchar(10) default NULL,
	tag varchar(10) default NULL,
	timestamp datetime default NULL,
	program varchar(15) default NULL,
	msg text,
	seq int(10) unsigned NOT NULL auto_increment,
	PRIMARY KEY (seq),
	KEY host (host),
	KEY seq (seq),
	KEY program (program),
	KEY timestamp (timestamp),
	KEY priority (priority),
	KEY facility (facility)
);</div>
</pre>
<p>Then, process those SQL commands using MySQL client:</p>
<pre># mysql -u root -p <syslog .sql</pre>
<h3>Setting up the communication channel</h3>
<p><em>syslog-ng</em> will issue INSERT INTO SQL commands into a UNIX pipe for every log received and processed. Those SQL commands will be retrieved from the UNIX pipe and will be injected into MySQL.</p>
<p>This UNIX pipe will act as the communication channel between <em>syslog-ng</em> and MySQL. To create the UNIX pipe:</p>
<pre>
<div># mkfifo /tmp/mysql.pipe</div>
</pre>
<p>Also, we will create a startup script used to keep feeding SQL commands sent to the UNIX pipe to MySQL called <code>/usr/local/etc/rc.d/040.mysql-syslog.sh</code>:</p>
<pre>
<div>( while [ -e /tmp/mysql.pipe ]
  do
        /usr/local/bin/mysql -u root --password= syslog </div>
</pre>
<p>This script will get invoked at startup and will keep feeding the SQL commands generated by the <code>mysql</code> <em>syslog-ng</em> destination into the MySQL database.</p>
<p>However, we must make sure this startup script is invoked <em>after</em> MySQL has been started. Thus, in FreeBSD I recommend renaming the MySQL startup script:</p>
<pre># mv /usr/local/etc/rc.d/mysql-server.sh \
     /usr/local/etc/rc.d/030.mysql-server.sh</pre>
<h3>Setting up syslog-ng</h3>
<p>Modify <code>/usr/local/etc/syslog-ng/syslog-ng.conf</code> to add a new source called <code>net</code> used to retrieve logs via the network:</p>
<pre>source net { udp(); };</pre>
<p>Next, add a new destination for MySQL:</p>
<pre>
<div>destination mysql { pipe("/tmp/mysql.pipe"
template("INSERT INTO logs (host, facility, priority, level, tag,
timestamp, program, msg) VALUES ( '$HOST', '$FACILITY', '$PRIORITY',
'$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC', '$PRORAM',
'$MSG' );\n") template-escape(yes) ); };</div>
</pre>
<p>Finally, configure <em>syslog-ng</em> so all logging is sent to the MySQL destination. Since <em>syslog-ng</em> allows multiple destinations, this makes perfectly possible to keep locally-generated log events stored in text files alongside the MySQL database.</p>
<pre>
<div>
log { source(net); destination(mysql); };
log { source(src); destination(mysql); };
</div>
</pre>
<p>Finally, we send the <code>SIGHUP</code> signal to <em>syslog-ng</em> to instruct it to re-read its configuration file and reconfigure accordingly:</p>
<pre># pkill -HUP syslog-ng</pre>
<p></syslog></pre>
]]></content:encoded>
			<wfw:commentRss>http://www.felipe-alfaro.org/blog/2005/12/21/configuring-syslog-ng-to-store-logs-into-a-mysql-database/feed/</wfw:commentRss>
		<slash:comments>1</slash:comments>
		</item>
	</channel>
</rss>

