Jun 27

Ubuntu Lucid Lynx (10.04) Fixup Script

GNU/Linux, Shell Scripts No Comments »

I wrote a super simple script to fix some issues I have with some defaults in the latest Ubuntu release.  Nothing majour here, just annoyances really.  Below is the script, read it and comment out any lines you do not want to take effect on your system. It requires root access for a few of the commands, such as managing packages and removing system-wide files. Run it as a regular user and the sudo command will be used to prompt you for a password, run like this:

$ chmod +x ubuntu-fix.sh
$ ./ubuntu-fix.sh

And the following is the source code or download the script here.

#!/usr/bin/env bash
 
#
# Simple script to remove some of what I consider to be annoyances with
# Ubuntu 10.04 Lucid Lynx (and probably future versions).
#
 
set -e
 
# keep location bar always visible instead of breadcrumbs
gconftool-2 --set '/apps/nautilus/preferences/always_use_location_entry' \
    --type bool 'true'
 
# move buttons to the correct side and add menu on left icon
gconftool-2 --set '/apps/metacity/general/button_layout' --type string \
    'menu:minimize,maximize,close'
 
# make fonts a reasonable size
gconftool-2 --set '/desktop/gnome/interface/font_name' --type string 'Sans 9'
 
# remove the poorly named and useless (to me) "ubuntuone" client package
if [ `dpkg -s ubuntuone-client | grep -c 'not-installed'` -ne 1 ]
then
    sudo apt-get remove --yes --purge ubuntuone-client
    sudo rm -f /etc/xdg/autostart/ubuntuone-launch.desktop 
fi
 
# remove the panel thing with email and other icons with wrong bg color
if [ `dpkg -s indicator-applet | grep -c 'not-installed'` -ne 1 ]
then
    sudo apt-get remove --yes --purge indicator-applet
    sudo rm -f /etc/xdg/autostart/indicator-applet.desktop 
fi
 
# add the regular volume icon back at startup
if [ ! -f /etc/xdg/autostart/gnome-volume-control-applet.desktop ]
then
    sudo cat /etc/xdg/autostart/gnome-volume-control-applet.desktop<<EOF
[Desktop Entry]
Name=Volume Control Applet
Comment=Show desktop volume control
Icon=multimedia-volume-control
Exec=gnome-volume-control-applet
Terminal=false
Type=Application
Categories=
NoDisplay=true
OnlyShowIn=XFCE;
X-GNOME-Bugzilla-Bugzilla=GNOME
X-GNOME-Bugzilla-Product=gnome-media
X-GNOME-Bugzilla-Component=gnome-volume-control
# See http://bugzilla.gnome.org/show_bug.cgi?id=568320
#X-GNOME-Autostart-Phase=Panel
X-GNOME-Autostart-Notify=true
X-GNOME-Autostart-Delay=2
X-Ubuntu-Gettext-Domain=gnome-media-2.0
EOF
fi
 
# offer to logout
echo -n "Done fixing Ubuntu, you need to log out and log back in for some of \
the changes to take effect.  
 
Log out now? [y/N] "
read DOLOGOUT
if [ ! -z $DOLOGOUT ] && [ $DOLOGOUT == "yes" ] || [ $DOLOGOUT == "y" ] || \
        [ $DOLOGOUT == "YES" ] || [ $DOLOGOUT == "Y" ]
then
    /usr/bin/gnome-session-save --kill
fi
  • Share/Bookmark
May 23

Low Level Disk IO in Managed .NET

.NET and Mono No Comments »

I’ve been messing around with disk I/O in C# lately, attempting to read/modify the boot sectors and such.  I’ve used the lower-level functions for accessing disks like files, from the Win32 API.  Most of the signatures were adapted from the pinvoke.net website as well as some examples from MSDN.

What I’ve done is made one file/class to hold all the unsafe/unmanaged code and another to wrap those functions up nicely in a .NET-style class named DiskStream.  The DiskStream class inherits from the Stream object and so exposes an interface similar to a FileStream.  You can open up a DiskStream and read, write, and seek it just like a regular file.

There are a few gotcha’s associated with performing disk access like this:

  • You need to access the disk in multiples of the sector size (ie. 512 bytes).
  • I don’t believe this will work at all in any Windows below Win2K.
  • Currently has odd behaviour on disks with mounted volumes, if there’s a mounted volume, it appears only the MBR can be written.  I’m going to fix this in a future version so that the volumes get locked and/or dismounted before hand.

So here’s the code for accessing the Win32 API from .NET. I think this is pretty useful, since I had a hard time tracking down this stuff and making it work. Here’s the first class, which I call DeviceIO:

using System;
using System.IO;
using System.Runtime.InteropServices;
using System.Threading;
using Microsoft.Win32.SafeHandles;
namespace DiskLib
{
    /// 
    /// P/Invoke wrappers around Win32 functions and constants.
    /// 
    internal class DeviceIO
    {
        #region Constants used in unmanaged functions
        public const uint FILE_SHARE_READ = 0x00000001;
        public const uint FILE_SHARE_WRITE = 0x00000002;
        public const uint FILE_SHARE_DELETE = 0x00000004;
        public const uint OPEN_EXISTING = 3;
        public const uint GENERIC_READ = (0x80000000);
        public const uint GENERIC_WRITE = (0x40000000);
        public const uint FILE_FLAG_NO_BUFFERING = 0x20000000;
        public const uint FILE_FLAG_WRITE_THROUGH = 0x80000000;
        public const uint FILE_READ_ATTRIBUTES = (0x0080);
        public const uint FILE_WRITE_ATTRIBUTES = 0x0100;
        public const uint ERROR_INSUFFICIENT_BUFFER = 122;
        #endregion
 
        #region Unamanged function declarations
        [DllImport("kernel32.dll", SetLastError = true)]
        public static unsafe extern SafeFileHandle CreateFile(
            string FileName,
            uint DesiredAccess,
            uint ShareMode,
            IntPtr SecurityAttributes,
            uint CreationDisposition,
            uint FlagsAndAttributes,
            IntPtr hTemplateFile);
 
        [DllImport("kernel32.dll", SetLastError = true)]
        public static extern bool CloseHandle(SafeFileHandle hHandle);
 
        [DllImport("kernel32.dll", SetLastError = true)]
        public static extern bool DeviceIoControl(
            SafeFileHandle hDevice,
            uint dwIoControlCode,
            IntPtr lpInBuffer,
            uint nInBufferSize,
            [Out] IntPtr lpOutBuffer,
            uint nOutBufferSize,
            ref uint lpBytesReturned,
            IntPtr lpOverlapped);
 
        [DllImport("kernel32.dll", SetLastError = true)]
        public static extern unsafe bool WriteFile(
            SafeFileHandle hFile,
            void* pBuffer,
            int NumberOfBytesToWrite,
            int* pNumberOfBytesWritten,
            int Overlapped);
 
        [DllImport("kernel32.dll", SetLastError = true)]
        public static extern unsafe bool ReadFile(
            SafeFileHandle hFile,
            void* pBuffer,
            int NumberOfBytesToRead,
            int* pNumberOfBytesRead,
            int Overlapped);
 
        [DllImport("kernel32.dll", SetLastError = true)]
        public static extern bool SetFilePointerEx(
            SafeFileHandle hFile,
            long liDistanceToMove,
            out long lpNewFilePointer,
            uint dwMoveMethod);
 
        [DllImport("kernel32.dll")]
        public static extern bool FlushFileBuffers(
            SafeFileHandle hFile);
        #endregion
 
    }
 
}

With that out of the way, I wrote a Stream class to make accessing the disk more “,NET-ish”. Here’s the code for the stream class:

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Text;
using Microsoft.Win32.SafeHandles;
using System.Runtime.InteropServices;
 
namespace DiskLib
{
 
    public class DiskStream : Stream
    {
        public const int DEFAULT_SECTOR_SIZE = 512;
        private const int BUFFER_SIZE = 4096;
        private string diskID;
        private DiskInfo diskInfo;
        private FileAccess desiredAccess;
        private SafeFileHandle fileHandle;
 
        public DiskInfo DiskInfo
        {
            get { return this.diskInfo; }
        }
 
        public int SectorSize
        {
            get { return (int)this.diskInfo.BytesPerSector; }
        }
 
        public DiskStream(string diskID, FileAccess desiredAccess)
        {
            this.diskID = diskID;
            this.diskInfo = new DiskInfo(diskID);
            this.desiredAccess = desiredAccess;
            // if desiredAccess is Write or Read/Write
            //   find volumes on this disk
            //   lock the volumes using FSCTL_LOCK_VOLUME
            //     unlock the volumes on Close() or in destructor
            this.fileHandle = this.openFile(diskID, desiredAccess);
        }
 
        private SafeFileHandle openFile(string id, FileAccess desiredAccess)
        {
            uint access;
            switch (desiredAccess)
            {
                case FileAccess.Read:
                    access = DeviceIO.GENERIC_READ;
                    break;
                case FileAccess.Write:
                    access = DeviceIO.GENERIC_WRITE;
                    break;
                case FileAccess.ReadWrite:
                    access = DeviceIO.GENERIC_READ | DeviceIO.GENERIC_WRITE;
                    break;
                default:
                    access = DeviceIO.GENERIC_READ;
                    break;
            }
 
            SafeFileHandle ptr = DeviceIO.CreateFile(
                id,
                access,
                DeviceIO.FILE_SHARE_READ,
                IntPtr.Zero,
                DeviceIO.OPEN_EXISTING,
                DeviceIO.FILE_FLAG_NO_BUFFERING | DeviceIO.FILE_FLAG_WRITE_THROUGH,
                IntPtr.Zero);
 
            if (ptr.IsInvalid)
            {
                Marshal.ThrowExceptionForHR(Marshal.GetHRForLastWin32Error());
            }
 
            return ptr;
        }
 
        public override bool CanRead
        {
            get
            {
                return (this.desiredAccess == FileAccess.Read || this.desiredAccess == FileAccess.ReadWrite) ? true : false;
            }
        }
 
        public override bool CanWrite
        {
            get
            {
                return (this.desiredAccess == FileAccess.Write || this.desiredAccess == FileAccess.ReadWrite) ? true : false;
            }
        }
 
        public override bool CanSeek
        {
            get
            {
                return true;
            }
        }
 
        public override long Length
        {
            get { return this.diskInfo.Size; }
        }
 
        public override long Position
        {
            get
            {
                long n = 0;
                if (!DeviceIO.SetFilePointerEx(this.fileHandle, 0, out n, (uint)SeekOrigin.Current))
                    Marshal.ThrowExceptionForHR(Marshal.GetHRForLastWin32Error());
                return n;
            }
            set
            {
                if (value &gt; (this.Length - 1))
                    throw new EndOfStreamException("Cannot set position beyond the end of the disk.");
                long n = 0;
                if (!DeviceIO.SetFilePointerEx(this.fileHandle, value, out n, (uint)SeekOrigin.Begin))
                    Marshal.ThrowExceptionForHR(Marshal.GetHRForLastWin32Error());
            }
        }
 
        public override void Flush()
        {
            // not required, since FILE_FLAG_WRITE_THROUGH and FILE_FLAG_NO_BUFFERING are used
            //if (!Unmanaged.FlushFileBuffers(this.fileHandle))
            //    Marshal.ThrowExceptionForHR(Marshal.GetHRForLastWin32Error());
        }
 
        public override void Close()
        {
            DeviceIO.CloseHandle(this.fileHandle);
            base.Close();
        }
 
        public override void SetLength(long value)
        {
            throw new NotSupportedException("Setting the length is not supported with DiskStream objects.");
        }
 
        public override unsafe int Read(byte[] buffer, int offset, int count)
        {
            int n = 0;
            fixed (byte* p = buffer)
            {
                if (!DeviceIO.ReadFile(this.fileHandle, p + offset, count, &amp;n, 0))
                    Marshal.ThrowExceptionForHR(Marshal.GetHRForLastWin32Error());
            }
            return n;
        }
 
        public override unsafe void Write(byte[] buffer, int offset, int count)
        {
            int n = 0;
            fixed (byte* p = buffer)
            {
                if (!DeviceIO.WriteFile(this.fileHandle, p + offset, count, &amp;n, 0))
                    Marshal.ThrowExceptionForHR(Marshal.GetHRForLastWin32Error());
            }
        }
 
        public override long Seek(long offset, SeekOrigin origin)
        {
            long n = 0;
            if (!DeviceIO.SetFilePointerEx(this.fileHandle, offset, out n, (uint)origin))
                Marshal.ThrowExceptionForHR(Marshal.GetHRForLastWin32Error());
            return n;
        }
 
        public int ReadSector(DiskSector sector)
        {
            return this.Read(sector.Data, 0, sector.SectorSize);
        }
 
        public void WriteSector(DiskSector sector)
        {
            this.Write(sector.Data, 0, sector.SectorSize);
        }
 
        public void SeekSector(DiskSector sector)
        {
            this.Seek(sector.Offset, SeekOrigin.Begin);
        }
    }
 
    public struct DiskSector
    {
        public const int DEFAULT_SECTOR_SIZE = 512;
        private long offset;
        private byte[] data;
        private int sectorSize;
        public int SectorSize
        {
            get { return this.sectorSize; }
            set
            {
                if ((value % 2) != 0)
                    throw new ArgumentException("Sector size must be a multiple of 2.");
                this.sectorSize = value;
            }
        }
 
        public long Offset
        {
            get { return this.offset; }
            set
            {
                if ((value % this.SectorSize) != 0)
                    throw new ArgumentException("Sector offset must be a multiple of SectorSize.");
                this.offset = value;
            }
        }
 
        public byte[] Data
        {
            get { return this.data; }
            set
            {
                if (value.Length != this.SectorSize)
                    throw new ArgumentException("Data length must be the same as SectorSize.");
                this.data = value;
            }
        }
 
        public DiskSector(byte[] sectorData, long sectorOffset, int sectorSize)
        {
            if ((sectorSize % 2) != 0)
                throw new ArgumentException("Sector size must be a multiple of 2.");
            this.sectorSize = sectorSize;
            if (sectorData.Length != sectorSize)
                throw new ArgumentException("Data length must be the same as SectorSize.");
            this.data = sectorData;
            if ((sectorOffset % sectorSize) != 0)
                throw new ArgumentException("Sector offset must be a multiple of SectorSize.");
            this.offset = sectorOffset;
        }
 
        public DiskSector(byte[] sectorData, long sectorOffset)
        {
            int sectorSize = DEFAULT_SECTOR_SIZE;
            this.sectorSize = sectorSize;
            if (sectorData.Length != sectorSize)
                throw new ArgumentException("Data length must be the same as SectorSize.");
            this.data = sectorData;
            if ((sectorOffset % sectorSize) != 0)
                throw new ArgumentException("Sector offset must be a multiple of SectorSize.");
            this.offset = sectorOffset;
        }
 
        public DiskSector(long sectorOffset)
        {
            int sectorSize = DEFAULT_SECTOR_SIZE;
            this.sectorSize = sectorSize;
            this.data = new byte[sectorSize];
            if ((sectorOffset % sectorSize) != 0)
                throw new ArgumentException("Sector offset must be a multiple of SectorSize.");
            this.offset = sectorOffset;
        }
    }
 
}

You might’ve noticed the DiskInfo class, I’m not going to post the code here, but I’ll provide a download link. It’s currently just a wrapper around WMI/Win32_DiskDrive, although I’m planning on removing the WMI code and replacing it with more platform invoke calls. I’ve also added a DiskSector struct to (attempt to) make accessing the disk using sectors easier. The DiskSector struct can be used with the ReadDiskSector(), WriteDiskSector() and SeekDiskSector() methods of the DiskStream class.

And finally a WARNING to anyone intending to use this code: messing around with disks like this is EXTREMELY dangerous and you can hose your whole system very easily. I HIGHLY recommend you play with this code within a Virtual Machine (VirtualBox, Qemu, etc) or on a separate system that you don’t mind losing.

It’s also important to note that this is very early code, and it has not at all been thoroughly tested in any way. I’ve used it to toggle some bits in the MBR, read the partition table, and to wipe a disk clean, and that’s the extent of my testing. Proceed with caution and at your own risk.

I’ll update this post as I improve the code here.

Here’s a bunch of source files which could be useful in Zip format.

  • Share/Bookmark
Jan 09

Python Whois

Python, Shell Scripts No Comments »

A little Whois reader I wrote while messing around with sockets in Python.  It doesn’t do any processing on the reply, it just returns the whole thing as a string.  It’s also missing any mechanism to determine which Whois servers to use for which TLDs.  But anyway, it’s a good start!

#!/usr/bin/env python
import socket
import select
 
PORT = 43
BUFSIZE = 1024
LINEEND = '\r\n'
 
def whois(domain, server, port=PORT):
    '''
    Perform a WHOIS search for domain on server/port.
    '''
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.connect((server, port))
 
    msg = ''
    lookup = domain + LINEEND
    readable, writable, error = select.select([],[s],[],60)
 
    if s in writable: s.send(lookup)
 
    readable, writable, error = select.select([s],[],[],60)
    while s in readable:
 
        data = s.recv(1024)
        msg = msg + data
        if not data:
            s.shutdown(socket.SHUT_RDWR)
            s.close()
            break
        readable, writable, error = select.select([s],[],[],60)
 
    else:
        s.shutdown(socket.SHUT_RDRW)
        s.close()
 
    return msg.strip()
 
if __name__ == '__main__':
 
    print whois('google.com', 'whois.markmonitor.com')
  • Share/Bookmark
Oct 11

Testing an LCD Monitor for Bad Pixels

Python No Comments »

I wrote a little cross-platform Python script to run a monitor through a sequence of colors to assist in detecting stuck pixels. The project is hosted here.

  • Share/Bookmark
Oct 05

Python Module for Geo-Coding/Geo-Location

Python, Shell Scripts No Comments »

I’ve written a very simple python module/script to help find postal code, city, province, latitude/longitude matching the supplied postal code, city, province, or coordinate.

Check out the Google Code Project page for more information.

You can checkout the source code by issuing the following command:

svn checkout http://python-geolocate.googlecode.com/svn/trunk/ python-geolocate-read-only
  • Share/Bookmark
Sep 29

Osmosis Won’t Read ‘planet-latest.osm.bz2′

Uncategorized No Comments »

I couldn’t get osmosis to read my OSM planet file, I even tried downloading another one, which of course took a long time. Then I found a post on the mailing list talking about the problem.

The solution is rather simple, just extract the file before passing it to osmosis to read, for example:

bzcat planet-latest.osm.bz2 | ./bin/osmosis --read-xml file=- --bounding-polygon file="country2pts.txt" --write-xml file="country.osm.EXT"

The error message I received was this:

kManager waitForCompletion
SEVERE: Thread for task 1-read-xml failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to parse xml file planet-latest.osm.bz2.  publicId=(null), systemId=(null), lineNumber=4313, columnNumber=13.
	at org.openstreetmap.osmosis.core.xml.v0_6.XmlReader.run(XmlReader.java:113)
	at java.lang.Thread.run(Thread.java:636)
Caused by: org.xml.sax.SAXParseException: XML document structures must start and end within the same entity.
	at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
	at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
	at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:391)
	at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1390)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.endEntity(XMLDocumentFragmentScannerImpl.java:878)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.endEntity(XMLDocumentScannerImpl.java:581)
	at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.endEntity(XMLEntityManager.java:1369)
	at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(XMLEntityScanner.java:1740)
	at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.skipSpaces(XMLEntityScanner.java:1469)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.seekCloseOfStartTag(XMLDocumentFragmentScannerImpl.java:1351)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:1286)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2723)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:624)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:486)
	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:810)
	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:740)
	at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:110)
	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1208)
	at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:525)
	at javax.xml.parsers.SAXParser.parse(SAXParser.java:392)
	at javax.xml.parsers.SAXParser.parse(SAXParser.java:195)
	at org.openstreetmap.osmosis.core.xml.v0_6.XmlReader.run(XmlReader.java:108)
	... 1 more
Sep 28, 2009 7:30:16 PM org.openstreetmap.osmosis.core.Osmosis main
SEVERE: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed.
	at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)
	at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:85)
	at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:30)
  • Share/Bookmark
Sep 27

Installing Mapnik and Friends on Ubuntu 9.04

GNU/Linux, Python 7 Comments »

I’m posting this for anyone who’s had a hard time getting Mapnik installed
along with some handy utilities on Ubuntu 9.04. Here we go.

Mapnik from Source

Update the package listing and then upgrade all packages:

apt-get update
apt-get upgrade

After that install the dependencies:

sudo apt-get install g++ cpp \
libboost1.35-dev libboost-filesystem1.35-dev \
libboost-iostreams1.35-dev libboost-program-options1.35-dev \
libboost-python1.35-dev libboost-regex1.35-dev \
libboost-thread1.35-dev \
libxml2 libxml2-dev \
libfreetype6 libfreetype6-dev \
libjpeg62 libjpeg62-dev \
libltdl7 libltdl7-dev \
libpng12-0 libpng12-dev \
libgeotiff-dev libtiff4 libtiff4-dev \
libcairo2 libcairo2-dev python-cairo python-cairo-dev \
libcairomm-1.0-1 libcairomm-1.0-dev \
ttf-dejavu ttf-dejavu-core ttf-dejavu-extra \
libgdal1-dev python-gdal \
postgresql-8.3-postgis postgresql-8.3 \
postgresql-server-dev-8.3 postgresql-contrib-8.3 \
libsqlite3-dev  \
subversion build-essential

Now we need to get the Mapnik source from the repository:

mkdir -v ~/src
cd ~/src
svn co http://svn.mapnik.org/trunk mapnik
cd mapnik

After that we can configure and build Mapnik:

python scons/scons.py configure INPUT_PLUGINS=all \
OPTIMIZATION=3 \
SYSTEM_FONTS=/usr/share/fonts/truetype/ttf-dejavu/
python scons/scons.py
sudo python scons/scons.py install
sudo ldconfig

Now we’ll do a quick test to see if Mapnik installed correctly:

python # will start the python interpreter
>>> import mapnik
>>> exit()

If the import mapnik line didn’t spit out any errors, everything is probably ok.


Setting Up the Database

First we need to tweak some configuration files. Open
/etc/postgresql/8.3/main/postgresql.conf with your favourite
text edit and modify the following lines:

shared_buffers = 128MB
checkpoint_segments = 20
maintenance_work_mem = 256MB
autovacuum = off

Next edit /etc/sysctl.conf and add the following at the end:

kernel.shmmax=268435456

Now to avoid a reboot, we’ll update the kernel like this:

sudo sh -c 'echo 268435456 >/proc/sys/kernel/shmmax'

Then restart the PostgreSQL server:

sudo /etc/init.d/postgresql-8.3 restart

The next thing we need to do is create the database. Substitute
“your_user” below with your user name. Note: If your user name is ‘user’ you might get an error from the commands below, I did when I tried. 

sudo -u postgres -i
createuser -s your_user
createdb -E UTF8 -O your_user gis
createlang plpgsql gis
exit

Now setup PostGIS on our database:

psql -d gis -f /usr/share/postgresql-8.3-postgis/lwpostgis.sql

And then change some permissions on the database:

echo "ALTER TABLE geometry_columns OWNER TO your_user; ALTER TABLE spatial_ref_sys OWNER TO your_user;" | psql -d gis

Now we’ll enable the intarray module:

psql gis < /usr/share/postgresql/8.3/contrib/_int.sql

Installing Some Extra Tools

The tools we’ll be installing here are useful for working with OpenStreetMap
data and also to facilitate generating maps with Mapnik.

OSM2PGSQL

This utility will import OpenStreetMap data files into the PostGIS database. First we’ll install some dependencies:

sudo apt-get install build-essential libxml2-dev libgeos-dev libpq-dev libbz2-dev proj

And then checkout the latest version from the repository:

cd ~/src
svn co http://svn.openstreetmap.org/applications/utils/export/osm2pgsql/
cd osm2pgsql
make

Mapnik-Tools

We’re going to put this right in our home directory:

cd ~
svn co http://svn.openstreetmap.org/applications/rendering/mapnik/

Setting up the World

Now we’re going to grab some boundaries data for the world.

cd ~/mapnik
wget http://tile.openstreetmap.org/world_boundaries-spherical.tgz
wget http://tile.openstreetmap.org/processed_p.tar.bz2
wget http://tile.openstreetmap.org/shoreline_300.tar.bz2

We’ll extract them each into ~/mapnik/world_boundaries:

tar xfv world_boundaries-spherical.tgz
rm -v world_boundaries-spherical.tgz
mv -v processed_p.tar.bz2 shoreline_300.tar.bz2 world_boundaries/
cd world_boundaries
tar xfv processed_p.tar.bz2
tar xfv shoreline_300.tar.bz2
rm -v processed_p.tar.bz2 shoreline_300.tar.bz2

Importing Some Data

The last thing we’re going to do is put some OpenStreetMap data into our database. You can either import the whole planet:

cd ~
wget http://ftp.heanet.ie/mirrors/openstreetmap.org/planet-latest.osm.bz2

But it’s a HUGE file, so if you just want to have a smaller part of the world,
you can use extracts. To create your own extracts, read this tutorial. Otherwise
grab an extract for your area of interest from somewhere (ex.CloudMade). You’ll want to download the file ending in .osm.bz2. In my example I’ll be using british_columbia.osm.bz2.

cd ~
wget http://downloads.cloudmade.com/north_america/canada/british_columbia/british_columbia.osm.bz2

We just need to import the data into the database now, for this we’ll use osm2pgsql which we installed earlier.

cd ~/src/osm2pgsql
./osm2pgsql --slim -d gis ~/british_columbia.osm.bz2

And there you have it. If you want to test this out from the command line, you
can use the last portion of this tutorial, or to use python, try the examples here.

Resources

Most of this tutorial was ripped directly off from the following sites:

  • Share/Bookmark
Sep 26

Mapnik Package for Ubuntu

GNU/Linux No Comments »

I made a Debian/Ubuntu package for Mapnik v0.6.1. There’s a 32-bit and 64-bit package. I’ve only tested it on Ubuntu 9.04, but it will probably work on Debian.

To install it download the package and type:

sudo dpkg -i mapnik_0.6.1-1_i386.deb

Then when it complains about dependencies, type:

sudo apt-get install -f

To install all the dependencies and finish the install. Sorry I don’t have a repository to put this package in.

I think if you use GDebi Installer it will do everything in one step.

Note: There are lots of dependencies (~500MB on a clean Ubuntu alternate install), I used those listed here. I tried removing all the -dev packages and anything else that looked like it was just for compiling, but it didn’t work.

UPDATE: I’ve also made an osm2pgsql package from the latest subversion copy. Download it here.

Enjoy

Download the Mapnik v0.6.1 x86 Package Here

Download the Mapnik v0.6.1 amd64 Package Here

  • Share/Bookmark
Sep 20

GPS Device Detection in Python

GNU/Linux, Python, Shell Scripts 2 Comments »

As part of one of my current projects, maptop, I needed a simple module to detect whether or not a GPS receiver is connected to the computer, and if so, which port is it connected to.

For the task, I used PySerial, along with some of their example code, to test if each serial port can be opened, and if so, if there are NMEA GPS sentences coming in on it.  It uses a fixed baud rate of 4800, which is what my receiver is using and is also specified here.  The timeout of 2 seconds seems to work (at least with my device).  Both the baud rate and the timeout can be changed by modifying the BAUD_RATE and TIMEOUT “constants” at the top of the script.

I’m not sure if this works on Windows or not, I haven’t tested it yet, but it *should work*.

Below is the code for the module:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88

#!/usr/bin/env python
# 
# Copyright 2009 Matthew Brush < mbrush AT leftclick DOT ca >
# 
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
# 
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
# 
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
# MA 02110-1301, USA.
 
__version__ = "0.1"
__all__ = [ 'get_gps_port', 'test_port' ]
 
import serial
import glob
import re
import platform
 
BAUD_RATE = 4800    # the standard nmea baud rate
TIMEOUT = 2         # time to wait for nmea sentences
 
class NoGpsReceiverError(Exception):
    def __init__(self, value):
        self.value = value
    def __str__(self):
        return repr(self.value)
 
def scan():
    """ Scan for available ports. return a list of device names. """
    if platform.system == 'Windows':
        import scanwin32
        ports = []
        for order, port, desc, hwid in sorted(scanwin32.comports()):
            ports.append(port)
        return ports
    else:
        import scanlinux
        return scanlinux.scan()
 
 
def test_port(port):
    """ Detects whether NMEA GPS sentences can be read from the port. """
    nmea_gps_pattern = '^\$GP[A-Z]{3},'
    gps_detected = False
    try:
        s = serial.Serial(port, BAUD_RATE, timeout=TIMEOUT)
        for i in range(1,10):   # try 10 lines in a row
            line = s.readline().strip()
            m = re.match(nmea_gps_pattern, line)
            if m:
                gps_detected = True
                break
        s.close()
        return gps_detected
    except serial.SerialException:
        return False
 
def get_gps_port():
    """ Looks for GPS data coming in on any serial port and returns the port name. """
    for port in scan():
        if test_port(port):
            return port
    raise NoGpsReceiverError('Unable to locate a suitable GPS receiver.')
 
def main():
    print "Testing ports to find GPS receiver..."
    try:
        if platform.system == 'Windows':
            print "  Warning: gpsdetect not tested in Windows yet."
        for port in scan():
            if test_port(port):
                print "Detected GPS receiver on '" + port + "'."
                found_gps = True
                break
    except NoGpsReceiverError:
        print "No GPS receivers were detected on the system."
 
 
if __name__ == '__main__': main()

The complete source can be downloaded here.

  • Share/Bookmark
Sep 15

Keeping USB Drive from Sleeping

GNU/Linux No Comments »

I finally got so fed up of my Samba shares not being accessible after a while that I did some digging around. One of my 3 external USB drives attached to my file server keeps going to sleep, which sounds like a good thing. What happens, however, is when I attempt to access the shares from my Windows machine, I get a “Not Accessible” error and I can no longer use any of my Samba shares.

My file server is running Ubuntu 9.04 and all the drives are mounted under /media, then each important directory from each drive is symlinked to a sub directory of /media/shared, and /media/shared is the Samba share. This is convenient because it allows my file server, which has 5 various sized disks and attachments, to appear as one huge drive on the network. I’m not sure if this is why the one sleepy drive is bringing down the whole share or not.

To resolve the issue, I keep having to SSH into my file server, run a command such as fdisk -l to access the drive(s) and then restart Samba. There is a noticeable delay when fdisk -l hits that one sleepy drive while it spins up. My super simple hack to prevent this drive from spinning down, even though I’d rather let it sleep and wake on its own, is this:

$ sudo crontab -e

Then I added the following line to the cron tab file:

5 * * * * fdisk -l >/dev/null 2>&1

This will issue the fdisk -l command every 5 minutes, dumping the fdisk output to neverland. The idea is to access the drive every once in a while to prevent it from going to sleep. I expect to do some tweaking on the 5 minutes part, in an attempt to run the job less frequently, but still enough to keep the drive awake.

It’s an ugly hack and I’m sure there’s something I could do with hdparm if I spent some more time on it, but whatever. It’s only me accessing the file server and the performance is not very critical.

If you do know of slicker way of doing this, drop a comment and let me know.

  • Share/Bookmark
Previous Entries

1,076 spam comments
blocked by
Akismet