- Home /
[iOS] App freezes when executing pose detection (PoseNet + Barracuda)
Hello. In our project we implemented pose detection function which is implemented using PoseNet + Barracuda. It works well on Android devices, but on iOS, when first starting pose detection, app freezes about 5~10 seconds, then it starts work. During freezing, The following error occurred in XCode output:
Metal: Compute shader (Conv.Conv2DKernelKxK_T16x16_R4x4_NHWC) has 256 threads in a threadgroup which is more than the supported max of 224 in this device
2021-01-26 14:13:40.428697+0900 bod****dev[2764:1014125] Compiler failed with XPC_ERROR_CONNECTION_INTERRUPTED
2021-01-26 14:13:40.513960+0900 bod****dev[2764:1014125] Compiler failed with XPC_ERROR_CONNECTION_INTERRUPTED
2021-01-26 14:13:40.544636+0900 bod****dev[2764:1014125] Compiler failed with XPC_ERROR_CONNECTION_INTERRUPTED
2021-01-26 14:13:40.544773+0900 bod****dev[2764:1014125] MTLCompiler: Compilation failed with XPC_ERROR_CONNECTION_INTERRUPTED on 3 try
Metal: Error creating compute pipeline state: Compiler encountered an internal error
(null)
What I have tried:
upgrade/downgrade Barracuda version from 0.7.1 preview ~1.2.1 preview
change training models
try to modify pose detection code, but does not work
The relevant code is as follows:
IEnumerator PoseUpdateNoTex(NativeArray<Color32> buffer, int width, int height, float secondsToWait)
{
Model _model;
if (Application.platform == RuntimePlatform.Android)
{
byte[] modelBytes = BetterStreamingAssets.ReadAllBytes(modelName + ".bytes");
_model = ModelLoader.Load(modelBytes);
}
else
{
_model = ModelLoader.LoadFromStreamingAssets(modelName + ".bytes");
}
var _worker = WorkerFactory.CreateWorker(WorkerFactory.Type.CSharpBurst, _model);
#if UNITY_EDITOR
var frame = new Texture2D(width, height, TextureFormat.RGB24, false);
frame.SetPixels32(buffer.ToArray());
frame.Apply();
#else
var bLot = false;
if (webcamTexture.videoRotationAngle == 90) {
bLot = true;
}
var _colors = GetRotatedColors(buffer.ToArray(), width, height, bLot);
var frame = new Texture2D(height, width, TextureFormat.RGB24, false);
frame.SetPixels32(_colors);
frame.Apply();
var t = height;
height = width;
width = t;
#endif
yield return new WaitForEndOfFrame();
int _Width, _Height;
if (width > height)
{
_Width = 257;
_Height = Mathf.CeilToInt(257f * ((float)height / (float)width));
}
else
{
_Width = Mathf.CeilToInt(257f * ((float)width / (float)height));
_Height = 257;
}
frame.ResizePro(_Width, _Height);
var inputs = new Dictionary<string, Tensor>();
var tensor = new Tensor(frame, 3);
inputs.Add("image", tensor);
_worker.ExecuteAndWaitForCompletion(inputs);
yield return new WaitForEndOfFrame();
var Heatmap = _worker.CopyOutput("heatmap");
yield return new WaitForEndOfFrame();
var Offset = _worker.CopyOutput("offset_2");
yield return new WaitForEndOfFrame();
var Dis_fwd = _worker.CopyOutput("displacement_fwd_2");
yield return new WaitForEndOfFrame();
var Dis_bwd = _worker.CopyOutput("displacement_bwd_2");
yield return new WaitForEndOfFrame();
poses = posenet.DecodeMultiplePosesOG(Heatmap, Offset, Dis_fwd, Dis_bwd,
outputStride: 16, maxPoseDetections: 1, scoreThreshold: 0.5f, nmsRadius: 20);
Offset.Dispose();
Dis_fwd.Dispose();
Dis_bwd.Dispose();
Heatmap.Dispose();
_worker.Dispose();
tensor.Dispose();
poseUpdated = true;
isPosing = false;
frame = null;
inputs = null;
yield return null;
}
I tried our app on several iOS devices, the errors are identical. I've been struggling this error for several days but found no clues. I'd appreciate it if anyone can help me. Thanks in advance.
My develop environment:
Unity 2019.4.16f1
XCode 12.3
iPhone 7: iOS 14.0.1
iPhone 12: iOS 14.3
iPhone 8: iOS 13.7
Barracuda: 1.0.2 (current)
Running into the same problem on my device. Looks like it has to do with Unity reporting the wrong maxComputeWorkGroupSize for certain iOS devices. Looks like the Barracuda team may have caught some, but not all of the devices. Added a ticket to the GitHub repo with all the useful info I could find: https://github.com/Unity-Technologies/barracuda-release/issues/150
Your answer
Follow this Question
Related Questions
The name 'Joystick' does not denote a valid type ('not found') 2 Answers
Bizarre Missing Assembly 'Common' Error 0 Answers
Unity Audio Record in iOS Problem 0 Answers
Qualcomm AR Projects 1 Answer