UTF-8 Encoding on Android

Question

UPDATE: i continued this question here.

I am not really sure what is the problem but here is the situation. I am trying to send a file to SoundCloud via HttpWebRequest. This is going fine in the Unity player on the pc, but on Android something goes wrong. I get the following error message when i run it on Android: Request entity contains invalid byte sequence. Please transmit valid UTF-8.

So, next thing i did was making sure i send in both situations (Unity-PC and Android) the same file, this did not work. Now i have added some code to write the bytes to a file.

 var bytesString = "";
 
             using (Stream s = req.GetRequestStream())
             {
                 foreach (MimePart part in mimeParts)
                 {
                     s.Write(part.Header, 0, part.Header.Length);
 
                     part.Data.Read(buffer, 0, buffer.Length);
 
                     while ((read = part.Data.Read(buffer, 0, buffer.Length)) > 0)
                     {
                         bytesString += new String(System.Text.UTF8Encoding.Default.GetChars(buffer));
 
                         s.Write(buffer, 0, read);
                     }
                     
                     part.Data.Dispose();
                     
                     s.Write(afterFile, 0, afterFile.Length);
                 }
                 
                 s.Write(_footer, 0, _footer.Length);
             }
             
             System.IO.File.WriteAllText(Application.persistentDataPath + Path.DirectorySeparatorChar + "test.txt", bytesString);
             Debug.Log ("Saved to " + Application.persistentDataPath + Path.DirectorySeparatorChar + "test.txt" );

In the image below you can see the result. On the left side the file from Android, on the right side the file from Unity on the pc. Can someone explain what is going wrong here? Let me know if you need more information.

alt text

Answer 1

Best Answer

Answer by renezuidhof · May 28, 2015 at 10:50 PM

It's working!! The problem was that at some point i used StringBuilder.AppendLine() to add a new line. This works fine on Windows, but on Android it didn't work... (i figured it out because the Content-Length was not the same for Windows and Android.)

I fixed it by instead of using 'StringBuilding.AppendLine()', i use 'StringBuilder.Append("\r\n")'

Add comment · Share

Answer 2

Answer by Bunny83 · May 23, 2015 at 05:05 AM

I just created a simple test project with almost exact the code of your other question. I tested it in the editor and on my Nexus7:

result Editor test

result Android test

Here's a hex dump of the part where the file data starts. On the left is the android on the right the PC test.

I actually used a socket server to receive the data on my PC. So i litterally used WWW to send a POST request to an actual socket (i used a simple TCP server program i made years ago in Delphi)

Here's the actual code i used in Unity. For some reason i couldn't switch "write access" from "Internal Only" to "External(SDCard)". Well i can switch, but Unity doesn't seem to save that setting at all ^^. If you close the player settings and reopen them it reverts back to "Internal Only" for me. Same for "Internet Access" (always "Auto"). Seems to be a strange bug in Unity. Have to test a bit more and have to file a bug report.

Anyways, to be able to actually read a file on my sdcard i added the permission manually to the manifest:

 <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />

And finally it worked. I placed the file at

 /sdcard/Test.wav

So on my device i just entered that path and the ip of my PC and the request arrived as expected.

So there has to be something else wrong on your side. If the screenshot is your actual wav file, it seems you're missing the wav header... Where does this file you want to send come from? Do you create that file yourself? Did you create and add the wav file header?

If you don't know how the header should look like, just use Google and you find things like this.

NOTE:
Keep in mind that in my test i used HTTP and not HTTPS. Since HTTPS is by far more complex since it's encrypted i can't get the raw date that easy. There might be some problems with HTTPS connections on Android. I've read a couple of times that Unity doesn't support self-signed certificates on Android. Could be that it's already fixed but at least it's worth checking. Do other HTTPS requests work fine?

Add comment · Show 8 · Share

Bunny83 · May 23, 2015 at 05:53 AM 0

Share

It looks like SoundCloud supports HTTP as well as HTTPS so you might want to try it without secure sockets. If all fails there are apps like tPacketCapture which can (like WireShark for PC) capture your whole network traffic. It seems it has some limitations (it doesn't support IPv6 according to some comments). However i used it a couple of times and just tried it with this request. I have found the request (though a bit fragmented) in the log and the file looks the same there.

renezuidhof · May 23, 2015 at 12:50 PM 0

Share

The file is indeed created by myself, i already use the code you send. I copied the .wav file that worked for Unity on my PC to my Android device to make sure it can't be the file that is the problem (or could it?).

I also tried HTTP ins$$anonymous$$d of HTTPS. This works fine for getting the SoundCloud token. When using HTTP for uploading the file i see somewhere in Wireshark the following response: '301 permanent redirect' to HTTPS. So this kinda sucks... I can't see the raw response SoundCloud sends me.

I mainly think the encoding is the problem because i get this response from SoundCloud: Request entity contains invalid byte sequence. Please transmit valid UTF-8

renezuidhof · May 23, 2015 at 12:52 PM 0

Share

Btw, requesting the SoundCloud token over HTTPS works fine for WebClient class and also works fine for the WWW class.

I've added the following to get that to work (insecure, i know)

 ServicePoint$$anonymous$$anager.ServerCertificateValidationCallback = (p1, p2, p3, p4) => true;

renezuidhof · May 23, 2015 at 02:07 PM 0

Share

I did requests to http://requestb.in/16x52sy1?inspect (Available from 48 hours)

These requests are done over HTTP.

The top one is from Android. The bottom one is from my PC.

Bunny83 · May 23, 2015 at 10:05 PM 0

Share

@renezuidhof:
Sorry, I've spent several hours and trying to figure out what the problem might be, but i can't find anything wrong. I've created an account myself, registrated my app and i can get an access token via direct method (user name and password) just fine.

What i've realised is that you're not adding your client id (or maybe you do?). Anyways even when adding it i always get the same error, just like you mentioned:

Request entity contains invalid byte sequence. Please transmit valid UTF-8

I guess it's a general purpose error message since it is returned even when you pass an empty byte array. I tried several different ways to issue the request. I even tried using URL parameters ins$$anonymous$$d of multipart post body data. In that case i didn't send any actual wav data and I actually get a different error message which was telling me that the audio file i submitted wasn't recognised as audio ^^. If i add the data within the URL (yes, i actually tried that as well) I get a "500 Internal server error" :D.

However it doesn't make sense to encode the file's data in the URL due to the length restriction (also URLEncoded the data is really huge ^^).

I checked the way Unity's WWWForm class composes the multipart data and everything seems to be correct (as stated in the RFC). I've looked at other C# implementations of multipart HTTP requests. The only difference i've found was that Unity doesn't include a "Content-Transfer-Encoding" header for each part of the multipart body. At least a file should have

 Content-Transfer-Encoding: binary

I even tried to inject this header into the post data, however the error stays the same. I've found a few posts of other users (not Unity related) which had the same problem, however no solution.

If this is an important feature for your application you might want to contact the SoundCloud support. Personally i think the documentation of their API is quite bad. They only provide examples for a few platforms and always using a premade SD$$anonymous$$ which hides how the actual HTTP requests should look like. A real technical documentation would be nice.

Anyways, i'm done with this API ^^

Show more comments

UTF-8 Encoding on Android

2 Replies

Your answer

Follow this Question

Related Questions