- Home /
UTF-8 Encoding on Android
UPDATE: i continued this question here.
I am not really sure what is the problem but here is the situation. I am trying to send a file to SoundCloud via HttpWebRequest. This is going fine in the Unity player on the pc, but on Android something goes wrong. I get the following error message when i run it on Android: Request entity contains invalid byte sequence. Please transmit valid UTF-8.
So, next thing i did was making sure i send in both situations (Unity-PC and Android) the same file, this did not work. Now i have added some code to write the bytes to a file.
var bytesString = "";
using (Stream s = req.GetRequestStream())
{
foreach (MimePart part in mimeParts)
{
s.Write(part.Header, 0, part.Header.Length);
part.Data.Read(buffer, 0, buffer.Length);
while ((read = part.Data.Read(buffer, 0, buffer.Length)) > 0)
{
bytesString += new String(System.Text.UTF8Encoding.Default.GetChars(buffer));
s.Write(buffer, 0, read);
}
part.Data.Dispose();
s.Write(afterFile, 0, afterFile.Length);
}
s.Write(_footer, 0, _footer.Length);
}
System.IO.File.WriteAllText(Application.persistentDataPath + Path.DirectorySeparatorChar + "test.txt", bytesString);
Debug.Log ("Saved to " + Application.persistentDataPath + Path.DirectorySeparatorChar + "test.txt" );
In the image below you can see the result. On the left side the file from Android, on the right side the file from Unity on the pc. Can someone explain what is going wrong here? Let me know if you need more information.
Don't you have a HEX-editor? Viewing binary data in Notepad++ makes not much sense ^^. It would be interesting to see the actual bytes.
The actual question however is: Why do you actually **decode** the bytes into UTF-8? If the inco$$anonymous$$g bytes is binary it makes no sense at all. UTF-8 is a text encoding format for unicode characters.
If you want to encode the inco$$anonymous$$g bytes as utf8 you have to use GetBytes. Though that wouldn't make much sense either.
$$anonymous$$eep in $$anonymous$$d that if an automatic encoding / decoding happens somewhere in C# the systems default encoding is used. For windows that's usually iso-8859-1 (but depends on your local settings). Android seems to use utf-8 as default encoding.
See this SO question.
If you use WWWForm to add binary data, this data isn't encoded at all. Unity will create a multipart HTTP request and the actual data will be sent as it is, as binary. This should work the same on all platforms. It makes no sense to encode binary data as any kind of text.
I put it in a byte-reader but i don't have a lot of experience with this, here are the results (if you need more, let me know):
I know that it doesn't make sense to open it in Notepad but for me it showed that there was difference between Windows and Android. So i thought this might give some information why. Again, i don't have a lot of experience with this :)
The problem here is that i want to know why it works on Windows and not on Android. I think it has something to do with the (default) encoding of the data. I don't really know how to fix it :(
I have tried the WWW and WWWform but i couldn't get that to work either
Well, "WriteAllText" uses the systems default encoding when writing the text to a file. However the string should be the same on all platforms. There's a version of WriteAllText that takes an encoding. However as i said it's pointless to interpret arbitrary binary data as text (no matter if you use UTF8, ASCII or ISO-8859-1). You actually try to convert the binary data into text which most likely will fail at certain byte combinations.
Use BinaryWriter(Flush() and Close() after) or FileStream.Write, you will save yourself a lot of lines of code you're doing.
Writing the bytes to a file is nog my main problem here, i just added it to try to see what happened. The main problem is, I think, sending data to the SoundCloud api that's not utf-8 (looking at the error message). I really appreciate your help BTW :)
Answer by renezuidhof · May 28, 2015 at 10:50 PM
It's working!! The problem was that at some point i used StringBuilder.AppendLine() to add a new line. This works fine on Windows, but on Android it didn't work... (i figured it out because the Content-Length was not the same for Windows and Android.)
I fixed it by instead of using 'StringBuilding.AppendLine()', i use 'StringBuilder.Append("\r\n")'
Answer by Bunny83 · May 23, 2015 at 05:05 AM
I just created a simple test project with almost exact the code of your other question. I tested it in the editor and on my Nexus7:
Here's a hex dump of the part where the file data starts. On the left is the android on the right the PC test.
I actually used a socket server to receive the data on my PC. So i litterally used WWW to send a POST request to an actual socket (i used a simple TCP server program i made years ago in Delphi)
Here's the actual code i used in Unity. For some reason i couldn't switch "write access" from "Internal Only" to "External(SDCard)". Well i can switch, but Unity doesn't seem to save that setting at all ^^. If you close the player settings and reopen them it reverts back to "Internal Only" for me. Same for "Internet Access" (always "Auto"). Seems to be a strange bug in Unity. Have to test a bit more and have to file a bug report.
Anyways, to be able to actually read a file on my sdcard i added the permission manually to the manifest:
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
And finally it worked. I placed the file at
/sdcard/Test.wav
So on my device i just entered that path and the ip of my PC and the request arrived as expected.
So there has to be something else wrong on your side. If the screenshot is your actual wav file, it seems you're missing the wav header... Where does this file you want to send come from? Do you create that file yourself? Did you create and add the wav file header?
If you don't know how the header should look like, just use Google and you find things like this.
NOTE:
Keep in mind that in my test i used HTTP and not HTTPS. Since HTTPS is by far more complex since it's encrypted i can't get the raw date that easy. There might be some problems with HTTPS connections on Android. I've read a couple of times that Unity doesn't support self-signed certificates on Android. Could be that it's already fixed but at least it's worth checking. Do other HTTPS requests work fine?
It looks like SoundCloud supports HTTP as well as HTTPS so you might want to try it without secure sockets. If all fails there are apps like tPacketCapture which can (like WireShark for PC) capture your whole network traffic. It seems it has some limitations (it doesn't support IPv6 according to some comments). However i used it a couple of times and just tried it with this request. I have found the request (though a bit fragmented) in the log and the file looks the same there.
The file is indeed created by myself, i already use the code you send. I copied the .wav file that worked for Unity on my PC to my Android device to make sure it can't be the file that is the problem (or could it?).
I also tried HTTP ins$$anonymous$$d of HTTPS. This works fine for getting the SoundCloud token. When using HTTP for uploading the file i see somewhere in Wireshark the following response: '301 permanent redirect' to HTTPS. So this kinda sucks... I can't see the raw response SoundCloud sends me.
I mainly think the encoding is the problem because i get this response from SoundCloud: Request entity contains invalid byte sequence. Please transmit valid UTF-8
Btw, requesting the SoundCloud token over HTTPS works fine for WebClient class and also works fine for the WWW class.
I've added the following to get that to work (insecure, i know)
ServicePoint$$anonymous$$anager.ServerCertificateValidationCallback = (p1, p2, p3, p4) => true;
I did requests to http://requestb.in/16x52sy1?inspect (Available from 48 hours)
These requests are done over HTTP.
The top one is from Android. The bottom one is from my PC.
@renezuidhof:
Sorry, I've spent several hours and trying to figure out what the problem might be, but i can't find anything wrong. I've created an account myself, registrated my app and i can get an access token via direct method (user name and password) just fine.
What i've realised is that you're not adding your client id (or maybe you do?). Anyways even when adding it i always get the same error, just like you mentioned:
Request entity contains invalid byte sequence. Please transmit valid UTF-8
I guess it's a general purpose error message since it is returned even when you pass an empty byte array. I tried several different ways to issue the request. I even tried using URL parameters ins$$anonymous$$d of multipart post body data. In that case i didn't send any actual wav data and I actually get a different error message which was telling me that the audio file i submitted wasn't recognised as audio ^^. If i add the data within the URL (yes, i actually tried that as well) I get a "500 Internal server error" :D.
However it doesn't make sense to encode the file's data in the URL due to the length restriction (also URLEncoded the data is really huge ^^).
I checked the way Unity's WWWForm class composes the multipart data and everything seems to be correct (as stated in the RFC). I've looked at other C# implementations of multipart HTTP requests. The only difference i've found was that Unity doesn't include a "Content-Transfer-Encoding" header for each part of the multipart body. At least a file should have
Content-Transfer-Encoding: binary
I even tried to inject this header into the post data, however the error stays the same. I've found a few posts of other users (not Unity related) which had the same problem, however no solution.
If this is an important feature for your application you might want to contact the SoundCloud support. Personally i think the documentation of their API is quite bad. They only provide examples for a few platforms and always using a premade SD$$anonymous$$ which hides how the actual HTTP requests should look like. A real technical documentation would be nice.
Anyways, i'm done with this API ^^
Your answer
![](https://koobas.hobune.stream/wayback/20220613200156im_/https://answers.unity.com/themes/thub/images/avi.jpg)
Follow this Question
Related Questions
Unicode Symbol Degree and Square 1 Answer
Unity AndroidBuild with UTF-8 C# Files 0 Answers
Why won't unity remote 4 work for me? 0 Answers
Setting airplane mode for android 1 Answer