koobas.hobune.stream
How do I make a simple POST request to Amazon S3?
Hello Unity Community, I am trying to have a web-build of a project put a simple .csv file onto our Amazon S3 bucket. The idea is that the web-player would gather some information during play, and then store that information in .csv format on the bucket that we have created on Amazon S3.
I am using the WWW class, as well as the WWWForm class to build an HTML form, and send that to Amazon's servers. The documentation for building the request form to amazon is here, the documentation for building the policy is here and the documentation on authentication and signing is here.
Below is the code that constructs the form:
using UnityEngine;
using System;
using System.Text;
using System.Collections;
using System.Security.Cryptography;
public class AWSWebFormBuilder
{
// TODO: still need to store the accessKeyID of the actual root account (mturk needs this and can't use amazon IAM).
const string rootID = "REMOVED";
const string rootAccessKeyID = "";
const string rootSecretAccessKey = "";
const string appAccessKeyID = "REMOVED";
const string appSecretAccessKey = "REMOVED";
const double requestValidForSeconds = 15.0f;
string CreateSignature(string date, string region, string service, string policy)
{
// Source: http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-UsingHTTPPOST.html
byte[] policyBytes = Encoding.Default.GetBytes(policy);
byte[] dateBytes = Encoding.Default.GetBytes(date);
byte[] regionBytes = Encoding.Default.GetBytes(region);
byte[] serviceBytes = Encoding.Default.GetBytes(service);
byte[] secretAccessKeyBytes = Encoding.Default.GetBytes("AWS4" + appSecretAccessKey);
byte[] requestBytes = Encoding.Default.GetBytes("aws4_request");
HMACSHA256 dateHash = new HMACSHA256(secretAccessKeyBytes);
dateHash.ComputeHash(dateBytes);
HMACSHA256 dateRegionHash = new HMACSHA256(dateHash.Hash);
dateRegionHash.ComputeHash(regionBytes);
HMACSHA256 dateRegionServiceHash = new HMACSHA256(dateRegionHash.Hash);
dateRegionServiceHash.ComputeHash(serviceBytes);
HMACSHA256 signingKey = new HMACSHA256(dateRegionServiceHash.Hash);
signingKey.ComputeHash(requestBytes);
HMACSHA256 finalSignature = new HMACSHA256(signingKey.Hash);
finalSignature.ComputeHash(policyBytes);
string signatureString = Convert.ToBase64String(finalSignature.Hash);
//Debug.Log(signature);
//string signature = BitConverter.ToString(finalSignature.Hash);
//signature = signature.Replace("-", "");
return signatureString;
}
public void BuildUploadRequest(ref WWWForm requestForm, string fileName, string fileContent)
{
DateTime currentTime = DateTime.UtcNow;
DateTime requestExpiryTime = currentTime.AddSeconds(requestValidForSeconds);
string iso8601ExpiryTime = requestExpiryTime.ToString("s", System.Globalization.CultureInfo.InvariantCulture);
string iso8601CurrentTime = currentTime.ToString("s", System.Globalization.CultureInfo.InvariantCulture);
iso8601CurrentTime = iso8601CurrentTime.Replace(" ", "").Replace("-", "").Replace(":", "");
string policy = "{ \"expiration\": \"" + iso8601ExpiryTime + "\"," +
" \"conditions\": [" +
" {\"bucket\": \"ourbucket\" }," +
" [\"starts-with\", \"$key\", \"Project 1 Output/\"]," +
" {\"x-amz-credential\": \"" + appAccessKeyID + "/" + currentTime.ToString("yyyymmdd") + "/us-east-1/s3/aws4_request\"}," +
" {\"x-amz-algorithm\": \"AWS4-HMAC-SHA256\"}," +
" {\"x-amz-date\": \"" + iso8601CurrentTime + "\" }" +
"]}";
policy = Convert.ToBase64String(Encoding.UTF8.GetBytes(policy));
string signature = CreateSignature(currentTime.ToString("yyyymmdd"), "us-east-1", "s3", policy);
requestForm.AddField("AWSAccessKeyId", appAccessKeyID);
requestForm.AddField("policy", policy);
requestForm.AddField("key", fileName);
requestForm.AddField("x-amz-credential", appAccessKeyID + "/" + currentTime.ToString("yyyymmdd") + "/us-east-1/s3/aws4_request");
requestForm.AddField("x-amz-algorithm", "AWS4-HMAC-SHA256");
requestForm.AddField("x-amz-date", iso8601CurrentTime);
requestForm.AddField("Signature", signature);
requestForm.AddBinaryData("file", Encoding.UTF8.GetBytes(fileContent));
}
}
Now, based on the documentation, that code should work. Unfortunately it does not. Whenever I try to make a POST request to Amazon's servers I get the following error:
403 Forbidden - SignatureDoesNotMatch - The request signature we calculated does not match the signature you provided. Check your key and signing method.
As the error suggests, I must not be generating my signature correctly. I've been trying to figure this out for quite some time now, and I'm at my wit's end as I don't really know what I'm doing wrong.
I've also tried resetting the access & secret access keys on the IAM console, but that didn't help.
Can anybody help me out?
One lead might be how I'm converting the strings in the 'CreateSignature' function into bytes. Since the H$$anonymous$$AC hashing functions need 'byte[]' as a parameter, I need to store the strings as a byte array. Unfortunately, amazon's documentation doesn't state how I should encode those particular variables. It does say to use UTF-8 for the policy, and Base64 for the final policy string and signature, but there is no mention on how to encode the other variables. I'm assu$$anonymous$$g this does matter, as the algorithm on their end might be using a different encoding method on those variables.
Well, it turns out following that particular documentation just doesn't work, and the server will always return a 403 Access Denied error. I found this article instead that suggested to use a SHA-1 algorithm instead of the SHA256 one, and the process of signing was just using your secret key instead of using a concatenation of keys as was outlined in the other documentation.
Once I used that algorithm (by actually using it, and specifying it in both the policy and the form), the server was now sending me actual useful errors. It turns out my format for time was also incorrect, and that my policy was already expiring before it even got to the servers, so I fixed that too.
The below code now works for me when I try to do a POST request to Amazon S3 (for uploading a file from a web build):
using UnityEngine;
using System;
using System.Text;
using System.Collections;
using System.Security.Cryptography;
public class AWSWebFormBuilder
{
// TODO: still need to store the accessKeyID of the actual root account (mturk needs this and can't use amazon IAM).
const string rootID = "REMOVED";
const string rootAccessKeyID = "";
const string rootSecretAccessKey = "";
const string appAccessKeyID = "REMOVED";
const string appSecretAccessKey = "REMOVED";
const double requestValidForSeconds = 60.0f;
string CreateSignatureSHA1(string policy)
{
byte[] policyBytes = Encoding.Default.GetBytes(policy);
byte[] keyBytes = Encoding.Default.GetBytes(appSecretAccessKey);
HMACSHA1 signHash = new HMACSHA1(keyBytes);
signHash.ComputeHash(policyBytes);
string finalSignature = Convert.ToBase64String(signHash.Hash);
return finalSignature;
}
public void BuildUploadRequest(ref WWWForm requestForm, string fileName, string fileContent)
{
DateTime currentTime = DateTime.UtcNow;
DateTime requestExpiryTime = currentTime.AddSeconds(requestValidForSeconds);
string iso8601ExpiryTime = requestExpiryTime.ToString("s") + "Z";
string iso8601CurrentTime = currentTime.ToString("s", System.Globalization.CultureInfo.InvariantCulture);
iso8601CurrentTime = iso8601CurrentTime.Replace(" ", "").Replace("-", "").Replace(":", "");
string policy = "{ \"expiration\": \"" + iso8601ExpiryTime + "\"," +
" \"conditions\": [" +
" {\"acl\": \"private\" }, " +
" {\"bucket\": \"ourbucket\" }," +
" [\"starts-with\", \"$key\", \"Project 1 Output/\"]," +
" [\"starts-with\", \"$Content-Type\", \"\"]," +
" [\"content-length-range\", 1, 102400], " +
" {\"x-amz-credential\": \"" + appAccessKeyID + "/" + currentTime.ToString("yyyymmdd") + "/us-east-1/s3/aws4_request\"}," +
" {\"x-amz-algorithm\": \"AWS4-HMAC-SHA1\"}," +
" {\"x-amz-date\": \"" + iso8601CurrentTime + "\" }" +
"]}";
policy = Convert.ToBase64String(Encoding.UTF8.GetBytes(policy));
string signature = CreateSignatureSHA1(policy);
requestForm.AddField("key", fileName);
requestForm.AddField("AWSAccessKeyId", appAccessKeyID);
requestForm.AddField("policy", policy);
requestForm.AddField("acl", "private");
requestForm.AddField("x-amz-credential", appAccessKeyID + "/" + currentTime.ToString("yyyymmdd") + "/us-east-1/s3/aws4_request");
requestForm.AddField("x-amz-algorithm", "AWS4-HMAC-SHA1");
requestForm.AddField("x-amz-date", iso8601CurrentTime);
requestForm.AddField("Signature", signature);
requestForm.AddField("Content-Type", "text/plain");
requestForm.AddBinaryData("file", Encoding.UTF8.GetBytes(fileContent), fileName, "text/plain");
}
}
I'm probably doing some redundant stuff by supplying both a "Content-Type" field, and supplying a mime-type parameter when uploading the binary data.
In any case, the above code works for me, and, after spending more than a week on this, I finally got this to work.
P.S. It's also not a good idea to openly store access keys in your code. I know I have them written in my above example, but this is still in-development. I will have to think of a security solution later (probably requires a webserver).
Thanks for this info! I'm suffering through this right now myself. It seems that SHA1 is no longer allowed for uploading to S3. It seems that before you could upload using SHA1 in some regions, for whatever reason. But not any more.
Do you have a follow up based on SHA64? I'll try to post anything I find out myself.
quick comments on this now that I have it working...
S3 seems to require "Signing Version 4" nowadays, which is more difficult due to it calling hashes with other hashes. Each stage of this key building is nonsensical from a human-readable point of view. However this page was useful as it gives the results of each hash at each step: https://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html
when running the form with UnityWebRequest make sure to call Post() AND Send()! Then when I tried to wait on the result (yield return www) that didn't work either! It returned immediatley with no result. I had to spin watching for www.isDone() ins$$anonymous$$d.
if you want this to upload to s3 during a build (not gameplay), then there is a S3 Upload plugin for Jenkins. I didn't play with it much but it might work. It might not. :P
