Shady Project: Category OSX

Upgraded to Leopard

Shady — Thu, 25 Oct 2007 16:19:00 -0700

And I am not a happy camper. Sure the eye candy is nice, and not everything I have set up broke but….

After doing an upgrade (in hindsight, not the best idea), my system bordered on unusable:

1.Journaling had to be turned off on the main hard drive, because mds and mdworker processes were using up to 90% of my CPU at any given time.

2.ATSServer was briefly using upwards of 50% of my CPU, but a reboot seemed to fix that.

3.My external firewire drive is not being recognized or mounted.

4.Launching the Intel Optimized version of BonEcho froze and crashed on numerous occasions.

5.ventrilo is broken

Even the unix side of things is a mixed bag: 1.postgresql is working

2.sml-nj seems to work

3.my custom compiled version of ruby seems to also work

4.apache/php is broken beyond repair

5.mysql is broken

6.nmap seems to work, as does MSF

All In all, this is pretty much the WORST upgrade experience I have ever had with any OS. It seems like I have no choice but to backup (good fucking luck since my firewire external isn’t being mounted) and do a format and reinstall.

I usually only associate that with windows. The more things change…

QuickSilver

Shady — Fri, 16 Feb 2007 07:15:06 -0800

I've only been using Quicksilver for a few days now, and I already can't imagine not being able to use it. Access to every remote corner of your mac with a few key strokes. Awesome.

Postgres Housekeeping

Shady — Mon, 29 Jan 2007 07:30:00 -0800

Previously, we installed postgresql on our OS X computers. Now lets do a little bit of house keeping and clean up to make it more convenient for general use.

We'll start with the easiest problem first - getting rid of that postgres user hanging around our login window. One simple command takes care of this:

sudo defaults write /Library/Preferences/com.apple.loginwindow HiddenUsersList -array-add postgres

You'll be prompted for your administrator password, and after this command executes, postgres will no longer show up in the login window. You will, however, notice an option "Other..." so you can still login as the postgres user if you wish. This will not affect your ability to use su or sudo and execute commands as the postgres user.

Next task is to get postgres to start up on its own. For this, we will use the newfangled technology provided by apple in 10.4: Launchd. Fortunatley, lots of smart (and not lazy) people have done the hard work for us. You can download a functional launchd item here

Note that the fine gentleman who made this Launchd item has his own tutorial for setting up postrgres. Check if out, if you like. There is a lot more detail to be had there, such as setting up perl and java to connect to postgres. However, his installation direction for postgres different then the one we used previously. As a result this we will need to change a few properties of this plist. An aside: since launchd items are just plists (xml files) we can use the plist editor that comes bundled with xcode tools to edit the file. And this works. A nice alternative, however, is the nifty little tool Launchd Editor, available here. Back to the action: we need to change the following properties of the plist file:

The value for -D should be /usr/local/pgsql/data (or where ever you set your data directory)
The value for log_directory should be /usr/local/pgsql/logs (or, again, where ever you set your logs directory).

Once we have made the proper changes, save the plist and fire up the terminal. We're just a few steps away from having postgresql run whenever we want it to. The following commands will finish this job for us:

sudo cp org.postgresql.postmaster.plist /Library/LaunchDaemons
sudo launchctl load /Library/LaunchDaemons/org.postgresql.postmaster.plist

At this point, launchd knows all about our new item, and all we have to do is tell it to start:

sudo launchctl start org.postgresql.postmaster

And with that, postgres should be up and running. Fire up Activity Monitor and check for the postgres process. You should see a few. If we want to stop the service, run the same command as above, but replace start with stop. Pretty straightforward. I like to set up two aliases in my .alias file for starting and stopping the service easily from the command line.

And there we are. We can nicely start and stop postgres, the user is gone from our window, and all is right with the world. If only we had some kind of GUI management tool for dealing with postgres . . . something like, oh, I dunno . . . .this, maybe?. Of course, if you're more partial to web based management solutions a la phpmysql, there is always phpPgAdmin.

Of course, we still only have on user set up . . . we have to use postgres to log in to our database. While I won't tell you exactly what you need to do in order to set up a new user, I will give you a hint: check the excellent postgres documentation for create role. With the documentation on that, and the GUI editing tool of your choice (or good ol' psql) you should have extra users up and running in no time. The LOGIN attribtue might help too.

Tune in next time, when we (finally!) get around to installing PostGIS.

PostgreSQL on OS X

Shady — Wed, 24 Jan 2007 21:00:00 -0800

Yesterday I rambled on about the benefits of RDBMS and spatial data. Today, we'll actually take the first step towards getting all that fun stuff working. Setting up Postgres is not really a difficult task, especially if you have ever built any unix software before. Let's get started.

A few notes before we get started: You will need to keep all this source somewhere on your machine while it's being built. Where you put this is up to you. I usually download everything to ~/Downloads, and extract the files into their own directories inside downloads, then cd to those directories and build the software from there. It makes it easier for me to see what I have downloaded, what I'm working on, and makes clean up of the sources after the build procedure easier. You can, of course, put the source wherever you want to. /usr/local/src is a popular place, so is ~/scratch. Whatever works for you.

Additionally, I will be installing all this software into /usr/local. This is the traditional location for add on software in a unix environment. The build scripts for postgres default to this location. If you wish to be more "mac-like" you could install postgres into, for example, /Library/PostgreSQL if you chose to. It's up to you. I'll show you how to change the location used for the installations. Note that it is important you remember where you install libraries and other software, because if you put things into a non standard location, you will have to tell the compiler where to look for and dependencies.

Postgres doesn't have a whole bunch of dependencies to worry about, unlike a lot of other complex unix-y type software. In fact, there really is only one prerequisite: readline.

Readline is a GPL library for doing fancy things with command histories, line editing, and other things. A lot of other software uses this library(ruby uses it as part of irb, for example), so installing it now will acutally help you in the future should you need to install any other software that relies on it. Steps for installing readline:

Download the source distribution from ftp.gnu.org/gnu/readline/readline-5.1.tar.gz using curl, wget, transmit, or you downloading program of choice.
Fire up Terminal.app (found in /Applications/Utilities) and change to the directory where you downloaded the file.
type in 'tar xvzf readline-5.1.tar.gz' sans quotes and hit return. This will extrat the soruce files for readline and put them into a directory
cd readline-5.1
Run the following command './configure --prefix=/usr/local' to run the configure script. The --prefix flag allows us to set the base installation directory for this software. If you wanted to put it somewhere else (not reccomended) you could specify that here.
Once configure is done running, run 'make' to actually build the software.
When all the output from make stops, run 'sudo make install' to install the newly built software.

And you're done! With readline, that is. The next step is to actually build postgres. The steps for this are pretty much the same as they are for readline; namely download, extract, configure, make, install. Instructions for postgres are as follows:

Download the current version of the postgres source (8.2.1 as of this writing). Note that you have two options for downloading the source: gzip and bzip. These instructions assume you get the tar.gzip version.
Open the terminal, navigate to the directory where you downloaded the source and run 'tar xvzf postgresql-8.2.1.tar.gz' to extract the postgres source to its own directory
cd into the postgresql-8.2.1 directory and run 'configure --with-includes=/usr/local/includes --with-libraries=/usr/local/lib' the --with flags tell the compiler where to look for the libraries postgre needs (like readline). Note that if you changed the --prefix when installing readline, you will need to put the appropriate directory in both --with-includes and --with-libraries. Similar to readline, the configure script for postgres will accept the --prefix flag, so if you wish to pust postgres somewhere different, you can specify that here.
Now that we have configured everything, we can run 'make'. When make is finished running, run 'sudo make install' to install postgre into /usr/local or other directory you specified in the configure step.

And thats pretty much it as far as compiling and installing postgres is concerned. However if you try to run postgres right now, you will be sorely dissapointed, for a variety of reasons. We still have a few more tasks we must complete before postgres is really up and running.

Our first priority is to set up a postgres user. The postgres daemon expects to be run as an unprivilged user. It also expects your data directory to be owned by this single user, and only accesible by this same user. So lets go ahead and create that user now. We do this in the same fashion as adding any other OS X user: System Preferences -> Accounts and click the little + to add a new one. You can name it whatever you wish, I chose postgres. Enter a good password for the user. Do not check 'Allow user to administer this computer'box. We want this user to have the minimum amount of privileges on our machine for security reasons. Note that this user will show up in the login screen like a regular old user. I'll tell you how to remove this account later.

Now that we have our postgres user set up, we can set up and initialize the data directory postgres will use to store our databases. Steps for initializing our data directory:

Open up Terminal.app and run the following command 'mkdir /usr/local/pgsql/data' This will create the actual directory in which our data will be stored. Note that if you installed postgres in a different location, such as /Library/PostgreSQL you can put the data directory in that location. In fact, you can really put this directory anywhere you want; /pgdata would work, or even ~/MyPostgreData.
run the command 'chown postgres /usr/local/pgsql/data'. This changes the ownership of the data directory we just created to the postgres user. Again, if you changed the directory, or the name of the postgres user, be sure to replace the username and path above with the values you used.

Now that we have created the user and the data directory, we need to initialize the directory. Postgres expects certain files and other things to be in this data directory, so we have to prepare it for use. In order to do this, we will have to login as the postgres user. However, we won't be logging out and logging back in, we will do this all from the command line:

Open up Terminal.app (you really probably should just be keeping it open)
Run the command 'su -l postgres' making sure to replace postgres with the username you chose. You will be prompted for the password. Enter the password you created when you set up the postgre user.
You are now logged in as the user postgres. Run the following command '/usr/local/pgsql/bin/initdb -D /usr/local/pgsql/data', again making sure to supply your prefix if you changed it from /usr/local during configuration.

Whether or not this succeeds will depend on which version of OS X you are running. Prior to 10.4, this will succeed and you will have postgres ready to run. However, if you have 10.4, this will fail with some cryptic error message about shared memory request failing due to due max size being smaller then the requested amount. This may seem daunting at first, be we can fix it.

It turns out that sometime in 10.4, Apple change the maximum amount of shared memory that an application could request. They also added some requirements as to how shared memory requests have to be configured. It all sounds complex, but the fix itself is relativley simple. Just follow these simple steps:

Open up Terminal.app
Run the command 'cd /etc' to change into the etc directory
Run the command 'sudo touch sysctl.conf'. This will create the file sysctl.conf and it will be owned by the root user.
We now have to edit the file we just created. Any plain text editor will do, i used command line vim, but you could use TextMate, TextWrangler, or even BBEdit. Add the following lines to this file:
- kern.sysv.shmmax=4194304
- kern.sysv.shmmin=1
- kern.sysv.shmmni=32
- kern.sysv.shmseg=8
- kern.sysv.shmall=1024
Each of these lines should appear on their own line
Now you will need to restart your computer to make these changes take effect
Once the computer has restarted, redo the steps above, starting with 'su -l postgres' and following through to '/usr/local/pgsql/bin/initdb -D /usr/local/pgsql/data'. This time, you should receive no error messages.

Okay, all thats left is to start up the postgres server and try to create some databases. Let's do that, shall we?

Do I even need to say it? Open up Terminal.app
Login as the postgres user 'su -l postgres' again if you used a different name be sure to use that one. Enter the password.
Run the command '/usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data -l logfile start'. This command will start the database server. The -D option will tell postgres where to find the data directory. The -l option will tell postgres where to put its logfile. We can set this to any location we desire, so long as postgres has write access. The above line will put the logfile in the current directory, and will name it logfile.
You should see the message 'server starting' and now, finally, postgres is up and running. Pat yourself on the back as necessary

Now that we have our server up and running, lets give it a test run shall we? Make sure that you have terminal up and running, and that you are logged in as the postgres user.

Run '/usr/local/pgsql/bin/createdb test'. This will, naturally, create a database. Name it whatever you want.
Run '/usr/local/pgsql/bin/psql test'. This connects us to our newly created database. The psql utilitiy allows us to execute SQL statements against the database we specify when we run the tool. Lets try to add some tables now.
Enter 'create table the_table (the_name varchar primary key, the_id serial);' into the pgsql window, sans quotes. This will create a table named the_table with two fields. You will get some NOTICE: output about the explicit sequence for the serial, and explicit index for primary key. Don't worry about those, it's just a notification that some extra database objects will be created, not a warning or failure.
Enter 'insert into the_table (name) values ('Bob');' into the psql prompt sans quotes. This will insert a new row into the table.
Now lets see if that insert actually worked. Enter 'select * from the_table;' without quotes, and we should get back a nice little table with our one row.

So it looks like everything is working as intended! Of course, we still have a few issues. Starting up and stopping the server is a pain, some postgres user is hanging around our login window, and we have to be logged in as that user to do anything useful. And we don't have a fancy GUI for managing our database. Not to worry. We'll cover all that stuff (and more) nex time. This post is already long enough, don't you think? Tune in next time for some postgres housekeeping tips and tricks.

RDMBS and Spatial Data on OS X

Shady — Tue, 23 Jan 2007 15:07:00 -0800

MySQL is nice enough for your average web application. Even larger applications can be run on it with the appropriate amount of tuning voodoo (see: wikipedia). There is even the MySQL MaxDB product (essentially SAPDB rebranded after its GPL release) available for Enterprise level MySQL, although this is not available for OS X.

Core data is also a nice solution for persistence inside a single application.

Oracle can also be run on OS X, provided you enjoy being turned upside down and shaken over a large funnel until all your money comes out. Not that this pertains specifically to Oracle on OS X, it's really more of an Oracle feature in general.

Each of these database server options are available for OS X. And they all have their various uses and proponents. So why am I here talking about Postgres? Well,

It's Free (speech, and beer).
It's enterprise ready (I seem to recall the .org primary name servers run on postgres)
PostGIS

For those who don't know, PostGIS is an addon for PostgreSQL that allows the use of geographic objects as native datatypes inside the SQL engine. This includes points, lines, polygons, as well as a variety of functions that can work with these data types to give us, say, area, intersection, and unions of the various objects. To use the industry parlance, PostGIS will "spatially enable" PostgreSQL which allows us to store a wide variety of geographic data. We can additionally point tools such as Quantum GIS, GRASS GIS, and even the various ARC tools at a PostgreSQL database for almost all of our spatial data storage needs. It is based on the Open Geospatial Consortium Simple Features Specification for SQL.

Storing spatial data in an RDMBS solves a wide variety of problems that happen when managing spatial data, especially with multiple users. We can manage concurrent access more easily, we have defined mechanisms for backing up the data, and we can even generate other file types (shapefiles, for example) from the data in the database. Any organiziation that has more then a few users working on large amounts of spatial data can benefit a lot through the use of a spatially enabled RDMBS

So why PostgreSQL then? Well, to my knowledge there is no equivalent library for MySQL (feel free to point my in the right direction if one does exist). Ditto for MaxDB. Oracle does have spatial extensions, but costs, to use a technical term, assloads of money. DB2 and MSSQL 2000 also have their own spatial extension, but fall into largley the same category as Oracle with regards to price. Although, there is a free spatial extension for MSSQL 2000 available, the price of the server itself is costly. Oh yeah, and neither one of those two run on OS X. PostgreSQL does.

Over the course of the week, I wil be writing more about setting up PostgreSQL on OS X (it's really not that hard), installing PostGIS, and getting everything working nice and happily. Time permitting, I will even experiment with the BaseTen framework. This library gives us all the ease of a CoreData-like API, with a PostegreSQL back end. Heck, with those two, we're halfway to a Cocoa Native GIS program. Wouldn't that be cool?

Stay tuned tomorrow, when I delve into the mysteries of PostgreSQL, OS X, and kern.sysv.shmmax.