Can a unity game be set to open files of a given extension?

Absolutely. Create a custom editor for the DefaultAsset type exactly the same way you make one for any other class- this overrides the inspector that's shown for anything that isn't handled explicitly by Unity already. By override OnInspectorGUI, you can write any editor code you like to determine the kind of object being inspected and display an appropriate inspector for that object type.

If you don't want to make a custom editor, but rather react to the file being double clicked in the project files in the editor, then just put the OnOpenAsset attribute on any static method with the signature "static bool OnOpenAsset(int instanceID, int line)" (the function name doesn't have to match, just the parameter types and return type). Within that method, you then have to determine if the object being opened is something you want to handle, which will probably involve turning the instance ID into the object itself with EditorUtility.InstanceIDToObject(), then AssetDatabase.GetAssetPath() on that object to get its path on the disk, and finally Path.GetExtension() with that path to gets its extension. If the extension matches what you want to handle, do what you need and return true to say "it's been handled", and if not then just return false.

Let me know if you need any additional information. I think I have a class laying around that lets you make custom editors for extensions explicitly and bypassing the need to add a bunch of additional clauses to the DefaultAsset inspector each time you want to be able to inspect additional asset types- it might be overkill for your needs since it uses custom attributes and finds all types in the loaded assemblies with that attribute, adds them to an internal collection, and sort of defers assets to those if they register to handle the extension(s). If you want it, just say the word and I'll go rip it out of whatever library I put it in.

UPDATE: It occurs to me I may have misunderstood your intent. If you're talking about a game build being able to open files in Windows that are double-clicked, I believe you can manage this with a combination of "file associations" (a simple registry key saying "open this file type with my application") and CommandLineArguments. You can find links and more information here.

However, this only works if you want to START your Unity game by double clicking the asset. If you're looking to pass an argument into an already running instance of your game, there's no real support for that. But, you can possibly cheat and pull it off (sort of) by allowing multiple instances of your game in the PlayerSettings, but having a startup script that immediately checks if another instance is running, and quits if one is. You can use the CommandLineArguments, applied to your file associations, to silently open a second instance of Unity in the background in this manner, change a PlayerPrefs value or assign a value to a local save file of some sort, and then quit. Your primary instance would have a function constantly checking that saved value to see if it's changed, then act appropriately when it is (check in OnApplicationFocus if you like, to cut down processing there). In this way, you may be able to pass a message to an open Unity game with the filepath of a save file.

Long story short- it's not trivial, and I'm not 100% sure it would work.