Storing images in the database – good or bad idea?

Tue, 26 Jun 2007 16:17:26 GMT

As many of you know, I write real estate web applications for fun and profit. I have to admit it really is an interesting area for innovative web applications. After all, writing a modern IDX solution requires taking advantage of AJAX technologies, mastering a web based mapping platform (like Microsoft Virtual Earth or Google Maps), integration w/ 3^rd party APIs and web sites (such as Zillow or Trulia), pretty dynamic SQL, and handling large amounts of graphical images. Needless to say, it can pose some interesting technical challenges at times.

I'm in the midst of an application architecture upgrade; I'm currently debating how I want store my MLS images. I currently store broker specific images in a database (currently just agent photos and property management photos). My biggest customer (Real Property Associates) has about 70 agent photos and 2000 property images stored in a database, (I currently use ASP.NET HTTP handlers to extract images out of the DB and re-size said image on the fly before I send the response stream to the browser). The NWMLS images are on the file system scattered in one of a thousand subdirectories.

Although this arrangement is more efficient than storing all 474,589 or so NWMLS images in a single directory, it's painful to deal with. Deleting photos for sold or expired listings is like going spelunking instead of doing a DELETE * FROM … WHERE … SQL query. SELECT queries are a lot more flexible that DIR commands. Modifying data only requires one set of DB permissions instead of setting file system permissions on a thousand directories. I don't have to worry about the physical location of my data, since it's just a SELECT query away (regarding of where SQL ends up putting it). Adding meta data is just an ALTER TABLE … ADD COLUMN … SQL command away. In short, even if SQL Server is slower than the file system, the benefits of dealing with it instead are worth the price, so far.

Conventional wisdom is that you don't store images in SQL server. Unfortunately, my good experience with smaller data sets is leading me to believe that conventional wisdom is wrong. Then again, there's a world of difference between 2000 images and a number hovering 25K shy of half a million images. In my case, the images tend to be smaller JPEGs (about 400 x 300 in size) usually hovering around 50K, and I suspect (though I don't know), as long as the size of the images are small-ish, the performance penalty you'll pay appears to be comparatively is minor. My DBA guru friends have told me as long as the image sizes are small, it should be fine. SQL finding a BLOB via a primary key should be pretty quick.

Of course, the only way to find out for sure is to go for it (and see if you regret it later). I suppose, I already know the answer I want to hear. I just want to feel good about it before I go ahead and database-tize my NWMLS image set. Rumor has it that SQL 2008 has a new FileStream datatype, that stores BLOBs in the file system, but yet remain an integral part of the database with transactional consistency, so perhaps I should wait? Ironically, doing nothing (and waiting for Microsoft to fix the problem) is sometimes the smartest design decision you can make.

So does anybody have any good or bad experiences with storing images in their DB / Web applications? How big of a dataset are you storing in the DB? How big of a dataset are you displaying in the web browser window? How do the MLS Vendors deal with this problem?

Is there a better way to compute a Median in SQL?

Tue, 01 May 2007 15:38:58 GMT

I'm currently playing with random ideas I have for Nimitz (aka the next version of my NWMLS search tool), and one of the things I'm thinking about is displaying the Median value of properties that match an arbitrary search criteria. I already do something similar to this now on Zearch (the current version of my NWMLS search tool). with MIN, MAX, and AVG. So I wrote a Stored Proc (using MS SQL 2005 features), and came up with the following. (FYI - I was inspired by this blog post).

   1:  CREATE  PROC [dbo].[GetMedian]

   2:  (

   3:      @Col varchar(100),

   4:      @Table varchar(100),

   5:      @SqlWhere  varchar(1000)

   6:  )

   7:  AS

   8:  BEGIN

9:

  10:      DECLARE @Count AS INT, @MiddleRow AS INT, @Median AS INT

  11:      DECLARE @SqlAll nvarchar(MAX)

12:

  13:      SET @SqlAll = 'WITH MLSSTATS AS '

  14:      SET @SqlAll = @SqlAll + '('

  15:      SET @SqlAll = @SqlAll + 'SELECT ROW_NUMBER() OVER (ORDER BY ' + @Col

  16:      SET @SqlAll = @SqlAll + ' DESC ) AS RowNum , '

  17:      SET @SqlAll = @SqlAll + '(SELECT COUNT(*) FROM ' +@Table + ' '

  18:      SET @SqlAll = @SqlAll + @SqlWhere + ') As Cnt,'

  19:      SET @SqlAll = @SqlAll + @Col

  20:      SET @SqlAll = @SqlAll + ' FROM ' + @Table + ' ' + @SqlWhere

  21:      SET @SqlAll = @SqlAll + ') '

  22:      SET @SqlAll = @SqlAll + 'select AVG(' + @Col +') from MLSSTATS '

  23:      SET @SqlAll = @SqlAll + 'where rownum  IN((Cnt + 1) / 2, (Cnt + 2) / 2)'

24:

25:

  26:      EXEC sp_executesql @SqlAll

  27:  END

So if, let's say, I wanted to find the median value of all the houses currently listed in the city of Redmond, I'd do something like this...

   1:  GetMedian 'ListPrice', 'MLSTABLE', 'WHERE City in (''Redmond'')'

And then the proc would do it's dynamic SQL black magic and end up executing this...

   1:  WITH MLSSTATS AS

   2:  (

   3:  SELECT ROW_NUMBER() OVER (ORDER BY ListPrice DESC ) AS RowNum ,

   4:  (SELECT COUNT(*) FROM MLSTABLE WHERE City in ('Redmond')) As Cnt,

   5:  ListPrice

   6:  FROM MLSTABLE

   7:  WHERE City in ('Redmond')

   8:  )

   9:  select AVG(ListPrice)

  10:  from MLSSTATS

  11:  where rownum  IN((Cnt + 1) / 2, (Cnt + 2) / 2)

The SQL is ugly, but that's that nature of the beast. Anyway, the problem is that I really want do to this...

   1:  SELECT AVG(listprice), MEDIAN('ListPrice', 'MLSTABLE', 'WHERE City = ''Redmond''')

   2:  FROM MLSTABLE

   3:  WHERE City = 'Redmond'

Or better yet, try to implement Median as a user defined aggregate function in C# / CLR language so it's even easier to use like so...

   1:  SELECT AVG(listprice), MEDIAN(ListPrice)

   2:  FROM MLSTABLE

   3:  WHERE City = 'Redmond'

According the above blog post it can't be done in a CLR language (which I don't understand). I guess user defined aggregates written in a CLR language only get the values of a set one at a time, instead of the entire set (which you would obviously need to compute a median)? Either way, I'm wondering if there's a way to convert my ugly proc into an ugly function instead? I think what's tripping me up, is that I need to define a variable that's accessible from dynamic SQL and returnable from the function that calls it.

SQL / T-SQL

Storing images in the database – good or bad idea?

Is there a better way to compute a Median in SQL?