koobas.hobune.stream
View Generated Source (After AJAX/JavaScript) in Unity
I am trying to download page source from a given site that heavily uses Ajax / Javascript to Generate the source. What I am trying to do is find a way to get the source downloaded after this occurs in Unity. I am familiar with doing so in a basic C# application using the web browser component or with HtmlAgility Pack LoadFromBrowser method. However, I have tried to add win forms into the application with no success as well as HtmlAgilityPack and neither of those work. When trying to add winforms to the application I get "Could not register the window class, windows error 0" error and a null reference exception. Same goes for HtmlAgilityPack as the LoadFromBrowser I believe uses the webbrowser component. Am I missing something with UnityWebRequest that could get the job done?
This isn't possible out of the box. AJAX and Javascript are client side things. So you would need a fully blown web client that actually supports interpreting and executing javascript. This has nothing to do with the HTTP protocol and just loading resources from a server. The HtmlAgilityPack is just an HTML parser, nothing else.
Note: When i said fully blown web client I did not mean the WebClient class. This is just a HTTP client. It does not interpret the content at all. What you would need is something like Chromium. There are already some implementations for C#, however they are mostly targetting WPF or WinForms applications. There are some solutions specifically made for Unity, but most are not free. Also note that the platform support is usually quite restricted.
Yeah I'm not looking for a solution for parsing or anything of that sort. I'm simply needing a way to wait for the Javascript / ajax to complete before I download the page source. Even if there was a way to get the web browser component in winforms to work in unity that would work fine as then I could parse what I want using HtmlAgilityPack. There is no other libraries that will load dynamic content that you know of?
