- Home /
WWW not getting site that doesn't end on .html
I am using WWW to download a web page. The problem is that right now, the website URL doesn't end in .html The URL is https://shopifycounter.monstercat.com/counter/realtime
Answer by Bunny83 · Oct 20, 2015 at 10:30 PM
Well, this is an endless event stream which uses a custom format. It doesn't represent a normal webpage. You can't use the WWW class here. WWW expects the stream to end and will only complete when when the stream finished. This stream is never closed. It periodically sends a "heartbeat" event to keep the connection open. You have to use System.Net
classes instead.
For example with such an helper class:
using UnityEngine;
using System.Collections.Generic;
using System.Net;
public class WebStreamReader : System.IDisposable
{
volatile bool running = false;
string url = "";
System.Threading.Thread thread = null;
Queue<string> buffer = new Queue<string>();
object lockHandle = new object();
~WebStreamReader()
{
Dispose();
}
public void Start(string aURL)
{
if (!running)
{
url = aURL;
thread = new System.Threading.Thread(Run);
thread.Start();
}
}
private void Run()
{
running = true;
ServicePointManager.ServerCertificateValidationCallback = (a, b, c, d) => { return true; };
var request = System.Net.HttpWebRequest.Create(url);
var response = request.GetResponse();
var stream = response.GetResponseStream();
byte[] data = new byte[2048];
while (running)
{
int count = stream.Read(data, 0, 2048);
if (count > 0)
{
lock(lockHandle)
{
string message = System.Text.UTF8Encoding.UTF8.GetString(data, 0, count);
buffer.Enqueue(message);
}
}
}
}
public string GetNextBlock()
{
string tmp = "";
lock(lockHandle)
{
if (buffer.Count > 0)
{
tmp = buffer.Dequeue();
}
}
return tmp;
}
public void Dispose()
{
running = false;
thread.Abort();
}
}
You can read the stream data using a coroutine like this:
public class ReadStream : MonoBehaviour
{
WebStreamReader request = null;
void Start()
{
StartCoroutine(WRequest());
}
IEnumerator WRequest()
{
request = new WebStreamReader();
request.Start("http://shopifycounter.monstercat.com/counter/realtime");
string stream = "";
while (true)
{
string block = request.GetNextBlock();
if (!string.IsNullOrEmpty(block))
{
stream += block;
string[] data = stream.Split(new string[] { "\n\n" }, System.StringSplitOptions.None);
stream = data[data.Length - 1];
for (int i = 0; i < data.Length - 1; i++)
{
if (!string.IsNullOrEmpty(data[i]))
{
string msg = data[i].TrimStart('\n');
int index = msg.IndexOf("dollarAmount");
if (index > 0)
{
msg = msg.Substring(index + 14);
int index2 = msg.IndexOf('}');
msg = msg.Substring(0, index2);
decimal amount = decimal.Parse(msg);
OnDataUpdate(amount);
}
}
}
}
yield return new WaitForSeconds(1);
}
}
void OnApplicationQuit()
{
if (request != null)
request.Dispose();
}
void OnDataUpdate(decimal aAmount)
{
Debug.Log("Received new amount: " + aAmount);
// Do whatever you want with the value
}
}
Each message seems to be in the format:
event: EVENTTYPE
data: DATA
The only two events i've seen is heartbeat
and update
. The heartbeat event has just "1" as data while update has a JSON string which looks like this:
{"sales":3502,"dollarAmount":17341.71}
edit
I added the method "OnDataUpdate" to seperate the whole parsing from the point where you can use the value. Since you only want the dollar amount value, that's what this method get passed as parameter everytime when a new update arrives, I used the "decimal" type since we seem to deal with currency here. The decimal type eliminates most errors that comes from a floating point representation of a number.
Note: It's extremely important that you call Dispose on the WebStreamReader object to stop the thread it started. If you don't do that the thread will keep running, even when you stop playmode in the editor. This will not only add a new thread each time you press play, but might also make that website refuse your new connection since the old one is still alive.
Is there a more simple way? I just need the Dollar amount, that's it.
Uhm, no. You can't just "load" a site here since it's an endless stream. Even the WWW class uses a thread behind the scenes. You can't avoid it when dealing with network connections. Since it's a stream it's possible that only a part of a message arrives and the other part arrives later. That's why most "protocols" which are embedded in an endless stream have some sort of "marker" to indicate when a new message begins. In this case it's simply two line breaks in a row. $$anonymous$$y example already gives you the stitched and cropped raw message. It also takes care of synchronising the threads.
About the parsing of the message that's up to you. If the data always just looks like this you can of course manually parse the number out of the string. I can add this to my example as well. I'll edit my answer.
This was really useful! Thanks! I've searched the internet for about two weeks for such solution.
Your answer
Follow this Question
Related Questions
UriFormatException: Invalid URI: Invalid port number for VirtualBox server using IPv6 Address 1 Answer
C# Unity function to escape URL querystring parameters 2 Answers
How to use POST service using www class? 0 Answers
Help searching google! 1 Answer
How to encode german umlaute when laoding local stored data via WWW class? 0 Answers