Arduino Unity3d FPS drop after twisting Potentiometer

Alright , so I saw plenty of stuff here on arduino-unity3d stuff. Attached , is my code:

Unity C# code:

 using UnityEngine;
 using System.Collections;
 using System.IO.Ports;
 using System;
 
 public class SphereController : MonoBehaviour {
 
     private SerialPort serialPort;
     public char portNumber;
     public float drag;
 
     // Use this for initialization
     void Start () {
 
         serialPort = new SerialPort("COM"+portNumber,38400);
 
         string[] portNames = SerialPort.GetPortNames ();
 
         foreach( string s in portNames)
             Debug.Log (s+"\n");
 
         serialPort.Open ();
     
     }
     
     // Update is called once per frame
     void FixedUpdate () {
 
         float angle=drag;
         try{
             angle = drag*(int.Parse (serialPort.ReadLine())-511)/512;
             
             serialPort.BaseStream.Flush();
 
         }catch(TimeoutException){}
 
         Debug.Log ("Angle =" + (float)((angle)));
         transform.rotation = Quaternion.Slerp(transform.rotation,Quaternion.Euler(0,angle,0),Time.deltaTime*3);
     
     }
 }

Arduino Code:

'#include

RunningAverage runningAverage(32);

void setup() {

pinMode(A0,INPUT);

Serial.begin(38400);

runningAverage.clear();

}

void loop() {

runningAverage.addValue(analogRead(0)); Serial.println((int)runningAverage.getAverage());

delay(20);

}'

Right , so the idea does is that the arduino sends potentiometer data to the Unity program via SerialPort. Then , that data is read by Unity , and the rotation of the sphere (the script is attached to a sphere) is controlled.

So, here's the problem. The program works fine until I interact with the pot. Say , the pot reading is 768 initially , I can see 768 being displayed on my console in UNity3D ; FPS is 160+. The moment I twist the pot , the FPS drops drastically and saturates to about 3-2 FPS X'(

Help me , please guys ?? :)