koobas.hobune.stream
Parsing and displaying KML
I'm trying to create a data visualization in Unity using kml data. Without worrying too much about what's in the kml files, I'm trying to figure out the best way to:
grab a number of large kml (10s to 100s of megs) files from various websites
Save the files to a datastructure (local files? a data object?) to make parsing more efficient
Be able to parse / search the kml data sets with different parameters, i.e. to find all instances of a tag (and grab attributes / child tags), search for a tag or attribute value, grab all tags within some distance of a coordinate, etc.
I'm curious if anyone else has worked on this or has ideas about the best way to go around it. The biggest consideration is always speed, of course, and I'm not a programmer by background, so while I could hack something together it would be a far-from-optimal solution.
I poked around on the Unity forums for an xml parser, and I've seen the csharp System.Xml.XmlReader (XmlTextReader, etc.) mentioned. It appears that there are different ways to access XML: from a text file or URI, from an xml node, from a dictionary, etc.
What would be the best (quickest) way to store and access the xml data without maxing out memory and bogging Unity down? Save it as a local text or binary file and parse as needed? Save it in memory as a node? Parse straight from the URI? Something else entirely?
Thanks much!
In case it helps anyone else: I've started out using an XmlDocument to manage the data in memory. With that, I can use xPath to traverse the tree and pull data as needed. I looked into XmlReaders, but they require you to look at the nodes in order rather than being able to jump around.
As for loading / unloading the xml from disk, a coworker suggested using a synchronized data stream, or a NoSQL implementation for very fast lookup without having to reload the entire DO$$anonymous$$. With NoSQL, as I understand it, I could keep a simple tree in memory and then store attributes and child nodes to disk as more or less a hash.
I'm still interested in hearing any other ideas folks have.
Hi how's the NoSQL working out . http://forum.unity3d.com/threads/38273-Lightweight-UnityScript-X$$anonymous$$L-parser
I'm about to start a similar project, well, I'm about to start to use unity on an old project where I need to read elevation data from google maps to create terrain - I want to keep the code small and this looks like the only way. I've looked at collecting my map data from other sources but the resolution is too low
I'd be really interested in any suggestions you have, thanks for sharing so far.
We've looked at google maps for terrain too, but as far as I know their elevation data is all as shaded relief maps, which means you can't extract an actual height out of it - have you found a way to do so?
For your case, I'm not sure how useful NoSQL would be - it sounds like you're going to be making requests and doing things with the responses, so there wouldn't be a lot of reason to save X$$anonymous$$L information to memory or disk. You'd probably have good luck with a very lightweight X$$anonymous$$L parser, and there are a few on the forums:
http://forum.unity3d.com/threads/38273-Lightweight-UnityScript-X$$anonymous$$L-parser
http://unity3d.com/support/documentation/$$anonymous$$anual/Reducing%20File%20size.html (look for $$anonymous$$ono.xml.zip on that page)
As you guessed not much luck, I was thinking to extract the elevation data and use the numbers not the images... a real head scratcher.... and then I looked a openmaps and wondered if I could spoof the terrain by following their colour key - plenty of coding at work at the $$anonymous$$ute so I'm going to shoot zombies for a few days and see if I wake up in the middle of the night with some solution.....
Texture is easy to get - you can use the google maps api to request it. As for the mesh, we're generating it entirely on the fly so I don't have anything to send you there, unfortunately. I can grab that portion of the topo image we created and give you code to turn it into an array of heights, if that works.
It's been a while since I asked this question, and in the meantime we've mostly solved the issue. I'd like to share our solution for anyone else that's trying to do the same thing.
We ended up using SharpKml (http://sharpkml.codeplex.com/), which is a .NET implementation of the entire KML 2.2 specification.
SharpKml is meant for .NET 4.0, so there are some issues to overcome, things like missing classes (Tuple) and methods (Enum.HasFlag). However, if you go through each of the errors with a google search, you will find ways to fake everything in .NET 3.5. I ended up adding a 2-tuple and 3-tuple implementation and some extension methods to get everything fixed. Once you do so, it works just fine in Unity.
Another issue is that you'll need the Ionic.Zip library to do kmz, and that library doesn't run nice in the webplayer - some issues with unmanaged code due to various encoding things. We were able to solve it, but that's an issue for another post :)
Once you get that all set up, then you just have to build a framework around that library that translates between the metadata within the KML (styles, etc.) to your Unity objects, and presto - you have support for KML. It's not easy, by any stretch of the imagination - it took a pro like me a few days to get it running, let alone create a framework - but it's doable and supports the entire KML specification.
Oh, and to address the original question, I should add:
Sharp$$anonymous$$ml does allow you to search through the $$anonymous$$$$anonymous$$L pretty efficiently, much more efficiently than traversing raw $$anonymous$$$$anonymous$$L. It builds class relationships (parent-child) that can be traversed with a simple recursive function. Also, an added bonus: the source code comes with a dozen examples, one of which is "open a $$anonymous$$$$anonymous$$L file and get all the placemarks."
It handles $$anonymous$$$$anonymous$$L and $$anonymous$$$$anonymous$$Z, even nested $$anonymous$$$$anonymous$$Z ($$anonymous$$$$anonymous$$Z with another $$anonymous$$$$anonymous$$Z inside it). You do have to handle downloading the $$anonymous$$$$anonymous$$L yourself, but by this point we have build a pretty robust download manager so that's not important. And it seems to deal well with large $$anonymous$$$$anonymous$$L.
Hi Julien,
I've successfully ported Sharp$$anonymous$$$$anonymous$$L to be used in Unity (works on PC and Android), however it gives us a aot fault when running on an iPad. I've replaced all Dictionary collections with a List/$$anonymous$$eyValuePair wrapped collection since someone told me there's an issue with Dictionaries. It didn't help.
Could it be that the reflection chokes on the getter/setters of the various kml elements? Possibly null getters? Have you by any chance tried your implementation on an iPad?
Any information on this would be much appreciated.
With regards, Bart Wttewaall
Hey @$$anonymous$$ediamonkey,
Unfortunately, we haven't run this particular application on iOS.
What are the specific errors you're seeing? What .NET compatibility level are you running?
These pages might help:
http://docs.unity3d.com/Documentation/$$anonymous$$anual/$$anonymous$$obileDotnet.html
http://docs.unity3d.com/410/Documentation/ScriptReference/$$anonymous$$onoCompatibility.html
Good luck!
Thanks for your quick response :)
I'm using .NET 2.0 (full) for both android and iOS. But that's not the issue. The problem seemed to be the use of reflection at runtime which caused the AoT errors.
I've located various places where PropertyInfo.GetValue was used to fetch the value of an Element type instance and replaced it with my own lookup method. I made a big, ugly if-else clause for every type of Element, cast the inco$$anonymous$$g object to the correct type and in a switch-case block returned the correct propery's value. I'm not done testing, but this approach seems to work well.
I'll post the code ones everything works on all platforms. Any other insights into AoT pitfalls are much appreciated.
Hi, @$$anonymous$$ediamonkey
I am making a GPS App using Unity and would like to use Sharp$$anonymous$$$$anonymous$$L. Does your code works for both android and iOS?
Thanks in advance.
