SiteGrid Update 2017/11/27

While it has been a long time, the project is not forgotten.  

Why has this taken so long?

Time and money! Currently, this is an all-volunteer product.  All of the raw data has been paid for by external sources, but there is far more processing that has to happen.  While all processing was done more than a year ago and ready to distribute the SiteGrid standard changed again and forced a reprocessing of the data.  This caused some problems because the SiteGrid calculator cannot handle the 234 MILLION records at one time.  After all, this creates a database bigger than 25GB. That said the push has started again.

Where are we?

  • Data has been moved to a new database container from the source DBF files that the shapefiles contained.
    • Formerly this had been in a series of MS Access databases by state and UTM Zone.
    • This was moved to a MySQL database server that and was then converted to a MariaDB database server.
    • The data has been moved again to an SQLite3 database file.  This structure makes it easier to move and process.  Most SQL processes do not time out and are solved by throwing more system resources at the problem.
  • The raw data was imported.  Previously only the processed data was put in a database.
  • The SiteGrid table was created adding the proper precipitation fields.  This was possible as the source data was able to provide monthly precipitation and so was then re summarized.

Where are we going?

While it would be preferable that we calculate all of the variables there is no documentation on how the SiteGrid values are calculated.  However, that said we are now working to estimate the East and West horizons.  Again there is no documentation on this process from the people that created the SiteGrid processing routine.  But they did let slip once that the system could only process a few hundred thousand points at a time.  This is not acceptable when it comes to 234 million records.  Currently, a routine to calculate the actual distance to the horizon and the angle will be written for all points.  This is another advantage of the SQLite database as many spatial component plugins and variants have been created for SQLite.

After all of the points are calculated it will then be necessary to tackle breaking the grid into pieces that the SiteGrid routine can manage and then spend many hours processing the additional components.

When will this be done?

Not to put too fine of a point on it.  But it will be done when it is done.  It is better to get it right than to get it fast.  After the processing is complete, the data will then be available for FREE to anyone how asks.