Simulator GIS – Darb Dabney’s OpenSim

Blemishes notwithstanding, nearly six months of back-burner work has reached a threshold of readiness and is outward bound to some engineering firms, flood mappers at FEMA, and interested parties within the county. A handful of known issues remain unresolved. Proper name is “tbsm45cm_20100823″, proper edition is “2010.08″.

This is the third version of the terrain. Second version was “2010.01″ and included multiple LiDAR data sets, but fewer than presently used, and was a topographic model only. First version was “2009.09″ and was mainly photogrammetry and FEMA LiDAR, and was the last version to be developed in California Coordinates. Once the massive NCALM LiDAR data sets were processed, it became easier to move everything into WGS84 UTM zone 10 north meters projection, WGS84 NAVD88 CONTUS Geoid 2003 vertical position.

The NOAA utility program VDatum, a brilliant Java-based application able to stream-process data sets of near-infinite size, brought the NCALM data to heel, and opened up decades of NOAA depth surveys to our use in integrated topographic-bathymetric surface modeling.

First-return NOAA ALACE LiDAR swaths were fused along the outer coast, as bare-earth filtered versions were not produced in 1997–2002; the benefits of LiDAR detail along the rocky coast do seem to outweigh the distracting appearance of structures near Rodeo Lagoon, Stinson Beach, and outboard Bolinas.

When ArcGIS 9.4 beta 2 reached its limit in ability to render the terrain dataset into 45cm grid over the full extent, the clipping quadrants created to resolve this problem ended up chopping a very small portion of Sonoma county that drains into Estero Americano; the full watershed remains intact in the 1-meter version of the terrain grid under analysis for county-wide hydrology. Likewise, the tighter clipping quadrants lost a few hundred meters of San Pablo Bay bathymetry just west of where Marin, Sonoma, Contra Costa, and Solano counties meet. Also, tighter clipping quadrants snipped a portion of the San Francisco Bar southerly of San Francisco’s Seal Rock that was intended to be part of the model. All of these areas exist in the 100cm grid, and will be part of drainage analysis.

Happily, we have updated the workstation to ArcGIS 10, and have been enjoying such great speed gains with Spatial Analyst that our ERDAS use has been noticably reduced. Finally, Spatial Analyst is often showing performance nearly on par with ERDAS. Thank goodness that the Raster Calculator survived the transition to version 10 ArcGIS!

Painfully, the existence of unutilized bathymetric data sets for upper broad-channel Corte Madera Creek and Bolinas Lagoon have been revealed this week. Hey, there’s already something to look forward to for the next build!

The new terrain is getting some immediate use in support of an effort to participate in ESRI’s Community Maps Program for large-scale topographic mapping. The Program provides a template geodatabase with 36 vector feature classes and two raster, into which local agencies may pour their data. Once tucked into a conforming schema, a template multi-scale map document is provided with 120 layers—30 at each of four large scales that correspond to Google Maps and Bing Maps projection and cache tiling schema. The difference is that the template document makes use of ESRI tools to allow much more local detail to be packed into a map designed with notably more sophisticated cartography than either Google or Bing maps now have. The Community Maps Program concept is that local agencies may publish their local detailed content in a fairly uniform style, while retaining a world-wide seamless context for their surrounding area.

Qualitatively, the effect is that, when viewing the ArcGIS.com topo map alongside either Google or Bing maps (on two monitors, with comparison made at the same scale), the ArcGIS.com map looks to be a larger scale. It isn’t, and I’ve measured the size of features to convince myself, but my mind insists that I’m zoomed farther in on the ArcGIS.com map for some reason. My guess is that it is a perceptual effect of the much greater amount of information that is cleanly displayed in the ArcGIS.com map versus the much sparser Google and Bing content at these large zoom levels. Try it out—it’s like a carto version of an optical illusion!

The 120 layers in the template large-scale topographic base map from the ESRI Community Maps Program are arranged to provide four precise cartographic designs for Google/Bing map cache levels 16 through 19, which correspond to these display scales
1:15000–1:6001 (level 16, a.k.a. ~9k)
1:6000–1:3501 (level 17, a.k.a. ~4.5k)
1:3000–1:1501 (level 18, a.k.a. ~2k)
1:1500–1:501 (level 19, a.k.a. ~1k)
One of the most attractive areas currently online is Toronto, ON where at levels 18 and 19, individual building outlines are graced with street addresses.

Anyhow, the new tbsm45cm model will serve County of Marin’s effort at large-scale topographic mapping several ways. First, it has made possible a very detailed hillshade that helps emphasize the grading around each hillside structure in the county. Second, it helps us to create the required metric topographic contours. These are necessary to meet world-wide mapping standards, and throughout this weekend, contours are being generated from a related (smoothed version) of the terrain on 50cm vertical interval. Needless to say, most of these won’t get used in the map renderings, but the ESRI cartographers have shared a very clever indexing scheme that will help us use this single set of metric contours to support the requirements for all four of our topographic map scales.

This edition of the county-wide terrain model has been upgraded several ways in terms of input data sets, extent on land, and integration of bathymetric soundings to eliminate coastline clipping.

The LiDAR data set from NCALM / GeoEarthScope has been reprocessed to use inidividual ground-classified returns rather than the 50cm gridded surface; this was made possible by NOAA VDatum, the Java application that has industrial-strength powers to position XYZ files.

Also, NOAA Airborne LiDAR for Assessment of Coastal Erosion (ALACE) data, which are first-return only from 1998–2002, were included where available along the Pacific ocean coastline and edges of some estuaries.

An Airborne 1 LiDAR data set was graciously provided by Marin Municipal Water District where it was available in the lower Lagunitas Creek drainage, and this was used where available and not overlapping with the higher-density NCALM data at Point Reyes Station.

More of the FEMA LiDAR data from Dewberry was included along the eastern, urban floodplain areas. In this terrain model, all FEMA points below 25 meters elevation NAVD88, OR points landing on areas with slope less than or equal to 11 percent, were included. In the 2010.01 terrain, only points below 50 feet NAVD88 were used.

NOAA depth surveys were included with very little filtering, for all nearby soundings since 1931—knowing that tidal bars are dynamic, but including all data as a starting point.

The California Seafloor Mapping Project’s phenomenal multibeam sonar work was incorporated from 2-meter grids offshore to California’s 3-nautical-mile limit, and 1-meter grids within west San Francisco Bay.

The earlier photogrammetric breaklines from VARGIS/Infotech were classified into ridge and road types, and only the road types were retained as hard breakline constraints. Ridge lines and water lines were retained as soft breakline constraints. This has mitigated some of the effects of ridge lines artifacts that derive from inconsistencies between the VARGIS photogrammetry work breaklines and contours, and between VARGIS breaklines or contours and overlapping LiDAR data sets.

Gridding the terrain dataset into a dsm, once again we are flirting with the limits of ArcGIS stability. The processing workstation is imaged with Windows Server 2003 to remove limits on output file size, and to permit reliable killing of wayward processes. (At this time, we’re not certain whether this constitutes a “top kill”, or a “static kill”, but in any case the process ends up terminated ;^)

The 2010.07 edition is a topographic-bathymetric surface model, and its prime use case is the generation of accurate synthetic drainage networks. Together, these features motivate a larger modeled extent than the 2010.01 edition. Along the northern area, the extent was grown to include all watersheds that touch or drain Marin County areas, with some clipping of the easternmost portions of the Petaluma River watershed. Offshore, the legal extent of Marin County guided the clipping area in San Pablo Bay; data extent from CSMP guided eastern limits at Richmond Channel, San Francisco’s North Beach and Presidio shorelines, and westerly from Seal Rock to the 3-nautical-mile limit, then northerly to include all of Bodega Head as imaged by NCALM data.

The extra area has made it infeasible to generate a single 40-cm grid as before, but we have cut the area into quadrants and have generated four tiles as 45-cm gridding. More news soon.

As I learned here, it is even easier than ever for Windows users to browse the Marin dsm40cm terrain.
All those keyboard shortcuts one may have learned to use Google Earth now work in Google Maps with the “Earth” button.
It’s a gas, and all the 40-cm detail is there.

I’ve tested it using Chrome browser on Ubuntu, and the Earth button does not show up.
But using Chrome on Windows XP, it’s there and it works pretty well. No need for composing a web site with the Earth plug-in. It’s just there in the default Google Maps on systems where the plug in would work.

For many of the world’s users of browser-based interactive mapping, integrated 3D viewing has arrived.
Check it out. Link provides an initial point near Civic Center, viewed in Google Earth through Google Maps.

Users who access Google Maps using Windows or Mac should be able to seamlessly launch the Google Earth plugin within Maps.
Just click the Earth button (should be up where the different views are in the upper-right part of the map) if it exists.
If it doesn’t exist, then you are using Linux or a smartphone or are in some other environment that doesn’t support the Earth browser plug-in.

My favorite Google Earth navigation shortcuts:

(Mouse control)
- Drag Mouse regular button to push globe around.
- Release while moving to let globe continue to move with intertia.

Mouse scroll one way or PageUp = pan forward fast
‘+’ = pan forward slowly
Mouse scroll other way or PageDown = pan reverse fast
‘-’ = pan reverse slowly
‘r’ (lowercase R) = return to vertical view, North upward: “get me out of this mess and back to Map view”

(keys used by themselves)
‘w’ or up-arrow-key = spin globe forward
‘a’ or left-arrow-key = spin globe left
‘d’ or right-arrow-key = spin globe right
‘s’ or down-arrow-key = spin globe back

(Ctrl- variants)
(two non-opposing directions at once are allowed)
Ctrl – ‘w’ or up-arrow-key = tilt view upward toward the zenith
Ctrl – ‘a’ or left-arrow-key = pan view left
Ctrl – ‘d’ or right-arrow-key = pan view right
Ctrl – ‘s’ or down-arrow-key = tilt view downward toward nadir

(Shift- variants)
(two non-opposing directions at once are allowed)
Ctrl – ‘w’ or up-arrow-key = orbit point-of-view downward around center view
Ctrl – ‘a’ or left-arrow-key = orbit point-of-view rightward around point in center view
Ctrl – ‘d’ or right-arrow-key = orbit point-of-view leftward around point in center view
Ctrl – ‘s’ or down-arrow-key = orbit point-of-view upward around center view

WILD and FAST
Use Mouse alternate button to free-form navigate
- where you “right” click on the map, a target appears for as long as you keep the alternate button down.
relative to the target:
- drag the cursor up to zoom out
- drag the cursor down to zoom in
- drag the cursor right to rotate view counter-clockwise
- drag the cursor left to rotate view clockwise.

And if everything is just too much, there’s always ‘r’ (lowercase-R) to set everything straight.

Enjoy!

It’s not fully 3D immersive, but hey, 2-1/2D ain’t half bad. The “dsm40cm” model of Marin County has been published as the county’s default terrain on Google Earth. It’s a great pleasure to work with folks who are not troubled by a county representing its surface on a 40cm single-precision float grid that weighs in at 77 GB. In terms of data bulk, that is about the same as the entire 30-meter version of the US National Elevation Dataset.

What one gets when piling that much detail into a single county of around 520 square miles of land area is every building pad, driveway, and crown of road paving that were resolved. The dsm40cm model was derived from an ESRI Terrain Dataset that incorporates our best available topographic contours (1:4800 scale 10-foot; 1:2400 scale 2-foot,) photogrammetric break and water lines, FEMA LiDAR and NCALM (GeoEarthScope) LiDAR data sets. The Terrain Dataset currently comprises 40 GB of vector GIS data.

When the finely detailed surface grids were first developed, we broke the county up into 20 work areas to maintain ArcGIS 9.3.1 in a stable and productive state, and 30cm posting interval grids were generated that covered the entire county–at least during development. When necessary, these grid tiles were mosaicked with ERDAS Imagine into a single seamless grid. The 40cm version was produced directly as a single seamless grid using ArcGIS 9.4 beta 1, on a workstation imaged with Windows Server 2003. The WGS84 UTM, NAVD88-Geoid 2003 result was provided to the Google Earth team earlier this year.

As with all GIS data sets, it seems, the more detailed it is, the more rapidly it may need updating. In the works for the next year or so are several improvements to the dsm40cm model. First: the photogrammetric break lines will be segregated into steeper sets that tend to run along ridges, and shallower slopes that tend to delineate road cuts and building pads. The ridge set will be used as soft constraints to resolve some artifacts where they rise above some contours.
Second: incorporate new LiDAR data as it becomes available. Some data has already been provided for the lowest part of Lagunitas creek, and it appears that Prof. Ellen Hines of San Francisco State University’s Department of Geography and Human Environmental Studies has been funded by USGS to gather LiDAR county-wide this year.

So there will be revisions, but an exciting aspect is to see data flows being brought into existence that support different levels of mirror world development.
Publishing the dsm40cm model in Google Earth is an important (and beautiful) threshold to cross. Making use of the dsm40cm model in county operations such as creek and watershed delineation will be the practical benefit that drives the work in the first place. And before too many more weeks, there may be entirely new approaches to publishing the data in an immersive environment (neither Second Life nor Opensim) to share.

Kent Woodlands building pad and driveway, in the shadow of Mt. Tam

Next week I plan to be on a panel at the Virtual Edge Summit 2010 in Santa Clara on Monday 22 Feb. Inspired by the style of a presentation by State of California GIO Mike Byrne at last month’s BAAMA.org educational session, I will try to prepare an Ignite-inspired talk for Virtual Edge. Since the virtual environment space almost demands it, I will use a film rather than a slide stack to structure my speaking.

The film is mostly cut, and a draft is uploaded in HD – for your perusal. If you want to hear my words try to keep pace with the film, consider attending Virtual Edge Summit, either in person or through one of its virtual channels. Hope to see you there!

If you have the bandwidth, please view the film in HD–I spent most of the past weekend cutting original FRAPS takes at full resolution so that the presentation could stand up to 1080p HD.

Many thanks to Singularity U, director Matt Rutherford, and to Randall Hand who brought it to my attention After chatting at SLCC 2009 this past summer, I appreciate the immediacy of this lecture. OpenSim is discussed around minute 37 (video is available at 720p HD, and is just over 51 minutes long.)
Discussion of augmented reality, and mirror world creation in Second Life and virtual world simulators, just after minute 44.

It’s hard for me to listen to the entire talk just one time and retain the best explanations – but clear and current they are. In a virtual environment, immersed in near-infinite possibilities, Rosedale may no longer be guiding the Second Life ship, but I believe he remains the compass needle

I don’t have much to say about these regions that hasn’t been written already, and my views have been less aesthetic than Shenlei’s.
But in the interest of boosting the bandwidth by which I can share OpenSim, I’ve invested in a much newer Adobe Premiere Elements than I’d been using for the past five or so years. It’s a gas to have it multi-thread while rendering, and I have direct-to-FLV write. Trying to share as much of the motion and fidelity via YouTube as I possibly can, I’ve crafted a video resolution that is a multiple of my Hippo / SL viewer screen. The FRAPS video direct to AVI (sorry, it’s Win XP) is 1600 x 1140 @ 10 fps. Yup, those are video frames. In the interest of surviving an upload, I’ve rendered them highly time-compressed, with output at 1515 x 1080 @ 15 fps. As of tonight there’s no sound, no intertitles, just the rushes.
oops, if I read the YouTube Instructions for best formats, I should have trimmed the width to 1440, which is a multiple of 16.
Also, I have more direct upload options now with Premiere 8 than I had with my (recently demised) copy Premiere 1.0. Go Figure ;^)
While the Windows box grinds out the video print, I’m over here on Ubuntu blogging in a tab of 64-bit Chrome 4.0.249.43 and it is fine & fast.

For these videos, I visited ScienceSim Geography22_44 region and set the view to wide angle, then sat up at about 500 meters and watched the regions rez their terrain. For some folks, it will rank right up there with watching lead-based paint dry. For geography folks I’m hoping that these few minutes of sped-up video will convey, by dogged repetition, the primacy of regions in the provision of virtual environment simulators.

By the way, I’ve got a task: I need to find a better buzz word for the GIS community. I’ve been advised by some serious and well-intentioned (not to mention well-informed) folk that terms like “virtual” and “immersive” are actually boring to GIS’ers. So I’ll need to think about how to convey the concepts of “Mirror World”, “Multiuser Virtual Environment”, “Immersive Connected Experience”, “Third-Person Virtual World”, and related concepts into a catchy moniker. Hopefully, one that is not presently trademarked, either!

I’m trying to remain serious about this, but some of the options are treacherous. Geography in Social Media has a possibly awkward acronym; maybe it can be saved in recognizable form as “GIS for Social Media” or “Geography for Social Media”: GFSM
The term “3D Map with Me” is terse, slightly ambiguous

Here is the video chopped as it was when uploaded with 1515 x 1080 resolution. Problem with that is that by not preserving dimensions at a multiple of 16, and saving my viewer’s aspect ratio rather than the (standard since 16mm film) 4:3 aspect, my upload is clobbered into something perhaps suitable for a smartphone. So please consider this the Smartphone Version of last night’s rushes:

Then, once again with feeling, or at least with a little more rest, there is what I hope to be an HD-friendly moving vision of OpenSim, as it appears on the ScienceSim Geography regions. Yes! After it ripened on the YouTube servers for a few hours, I now see all the higher-res versions available. At 1440 x 1080, this is pretty close to what I see on my screen with a live Hippo viewer.

And after a day’s cogitation: anyone care to comment on the term: “Social Immersive Media GIS” as a moniker? Oops — I used “immersive” 8^(

The work on Point Reyes Station has more LiDAR data on tap, but this evening I’m taking a solar sauna in the Kepler planetarium. It’s a captivating display for both young and older eyes.

A large planetarium has our solar system displayed dynamically

Just time to post a couple of shots from Alaska North in ScienceSim.com grid. This build is being created with resources provided by the ScienceSim Land Grant Program, and should remain accessible until June 2010. Downtown area has been refined with 21cm LiDAR grid sculpty prims.
I haven’t figured out how to get the high-res shots with Hippo Viewer, so the 7 Mpel camera in the Linden Lab viewer is still my favorite. Hippo seems to top out at 2 Mpel right now.

Alaska North in Science Sim, featuring 1:4 Point Reyes Station

21cm LiDAR grid (2007) draped with 10cm orthoimagery (2004)

View across Lagunitas Creek toward community of Point Reyes Station

Good stuff has been happening – and GIS data has been finding its way into OpenSim!

Thanks to the persistence of Kim Smith and with help from John Jainschigg, I made my first-ever live audience presentation in the World2Worlds venue on 2009 12 15. Trying to fit everything into a regular work day, I was fortunate to get support to do the presentation from my desk at work rather than from home–and I indulged in a new digital-USB headset for the occasion. Oddly, the headset seemed invisible on the podium ;^)

Speaking at Smarter Technology, 2009 12 15 -- John Jainschigg built a custom podium several steps tall so that people could see my tiny avatar. Photo courtesy of Chimera Cosmos

After nearly a year away from pushing OpenSim (corresponding with my first year on a new RL job), I was a bit nervous about having up-to-date stuff to offer the audience. So instead of a slide stack, I presented the machinima of models that I had brought from OpenSim into Second Life.

Lazy of me, perhaps, but the video did help create a certain party atmosphere in the airy auditorium at W2W. It also helped echo my emphasis on a certain use of OpenSim that I’ve been striving towards since October 2007.
But as so often during RL conferences, the very best part of the presentation experience is the (sometimes serendipitous) human connections that take place around it, and this presentation was no exception to that rule.

One audience member was Richard Hackathorn, who described Second Life in the context of urban planning for listeners to Arina Hadich’s Urban Design Podcast. His enthusiasm for my subject matter at the Smarter Technology presentation has led to an upcoming podcast at UrbanDesignPodcast.com (details to follow). Although I haven’t been a regular listener to podcasts, when the subject of the talks are focused on an area of interest, I am impressed by how much information can be conveyed in the time it takes to listen. The podcast seems to have more detail per minute than I can get reading a browser on my phone, certainly more than I can browse while driving, (and far more detail than I seem to be able to convey in a minute spent writing a blog ;^)

Another audience member was Shenlei Winkler of Fashion Research Institute, who is surely among this world’s most prim-prolific individuals. I’ve been reading about the various Shengri La regions since 2008, created by Shenlei in collaboration with IBM researchers. Ironically, I was collaborating with different IBM researchers about the same time, in 2008 03. These days, Shenlei appears very active with several types of support for the IEEE/ACM-hosted, OpenSim-derived grid known as ScienceSim.com. Shenlei very kindly took time to contact me after the Smarter Technology presentation, introduce me (with voice chat) to ScienceSim and its resident researchers, and encourage me to participate in the Science Sim Land Grant program. More details are described in paragraphs below.

Through ScienceSim, I’ve had the pleasure of interacting with Mic Bowman of Intel Labs, the group managing ScienceSm servers. The way that these folks have configured the servers is glorious. I’ve been a fan of running OpenSim on 64-bit Ubuntu with Mono since 2008 07, and the Intel crew have taken it to such another level that I find it astounding. In ScienceSim, it appears that the ODE physics engine runs for as little CPU cost as that which I’d experienced before with the near-trivial basicphysics. It is a rare treat for me to speak with configurators of OpenSim, much less those who strengthen and extend the simulator code!

Since I was firing up the forges to create some LiDAR sculpties for my ScienceSim project, I decided to warm up the works by creating some carefully-scaled terrain for ScienceSim’s Yellowstone 16-region model. Using public terrain data from the US Geological Survey, I processed terrain down to a model that was 1:83 in the x and y dimensions, and 1:55 in the z dimension (vertical scales are often exaggerated in both hominid avatars and terrain to make them more attractive). As a by-product of the production process, I also saved an intermediate bit of data that I had scaled to 1/10.38 in x and y and 1/6.9 in z. Knowing that it would present a worthy challenge to OpenSim server jocks, (the 1:83 model fit into 16 regions, but the 1:10 model would require exactly 1024 regions) I passed it to Mic along with the 1:83 terrain.

My jaw was somewhat slackened when, less than a week later I heard from Mic that much had been done with the
1:10 terrain. The Intel crew had actually had the tenacity to wait while all 1024 regions were brought up on a single processor; shortly afterward they prudently fit the regions onto a single blade, dual quad-core (Xeons?) system with 11 GB of memory and my favorite X86_64 Ubuntu/Mono environment. As if that wasn’t enough, they did this while configured with ODE physics! I’d say their effort was Olympian, but heck, it actually took them less than six days!

It’s been tremendous fun this month watching OpenSim make bold moves in the direction needed to support the sort of civic paraverse / Immersive Connected Experiences that could back-end many aspects of local agency operations. I’ve been looking for ways to get here since 2006 11, but with Intel’s demonstration of 1K regions on 1 dual-quad-core Xeon blade, I won’t look like such a fool scoping out costs for a county that would require 20K regions to build out at nominal 1:1 scale. (By contrast, in 2007 I had estimated that Linden Lab hosting of 575 regions would cost the City of Berkeley upwards of $60,000 per month!)

The Yellowstone work by Mic has been beautifully documented by Shenlei here, and here and here and here. I’m even using one of Shenlei’s Yellowstone sunset shots as my (Ubuntu) desktop background ;^)

More to the point of my post’s title, I have taken some outstanding publicly available LiDAR data and am in process of crafting a 1/3.9-scale model of Marin County’s own Point Reyes Station, a rural community that is situated so close to the San Andreas Fault that they were imaged in excellent detail as part of a scientific investigation into the geomorphic expression of strike-slip faulting. Together with publicly available high-resolution orthophotography, I’ve been able to refine the technique that was used on the 40-region model of the UC Berkeley campus and develop 21-cm-gridded surface model with 10-cm natural color orthophotography. For a preview of the model at 105-cm surface gridding, view this video. Enjoy, and watch for updates soon.

The Point Reyes Station may be trod upon in ScienceSim’s Alaska North region. Hope you get a chance to visit!