Rifle Accuracy w/ Rotation

I'm trying to make my rifle less accurate if the person is looking down a scope, and obviously more accurate when the player looks down the sights. I've been searching all over the forums and Unity Answers for a solution to this like FPS Gun Accuracy & Gun Accuracy with Random Range (which unfortunately are in JavaScript), but nothing I've tried has worked, or has gotten errors. The bullet has a script attached to it, controlling it's speed and other non-relevant information, and, it's rotation is based off the gun's shootpoint's rotation. I was thinking that I could change accuracy by changing the shootpoint's rotation very slightly ever time the player fires a bullet, making each bullet have a slightly different rotation. When the player looks down the sights, the amount the rotation changes is lowered, therefore making it more accurate. It should work, but I have no idea how to change the shootpoint's rotation. I thought the that I would have it's rotation in update, and then add a random range to it like this :

 accuracyModifier = Random.Range(minAccuracy, maxAccuracy);
 primarySP.rotation = Quaternion(1 + accuracyModifier, 1 + accuracyModifier, 1 + accuracyModifier);
 

But, that wouldn't work, because the rotation wouldn't always be 1, so I'm absolutely confused. I can't change the direction of the bullet because it's controlled by another script. Here's the script I use on the rifle (doesn't have accuracy modifiers) :

 using UnityEngine;
 using System.Collections;
 
 [ExecuteInEditMode]
 [AddComponentMenu("Void Runners/Weapons/Guns/Assault Rifle")]
 
 public class AssaultRifle : MonoBehaviour {
     
     public bool on = true;
     
     public string rifle;
     
     public float fireRate = 0.2F;
     public int clips = 5;
     public int roundsPerClip = 30;
     static public int clipsLeft;
     static public int roundsInClip;
     
     public GameObject round;
     //public AudioClip shot;
     
     private Transform primarySP;
     private Transform gun;
     
     private float nextFire = 0.0F;
 
     private bool reloading = false;
     private float bulletSpeed;
     
     void Awake () {
         
         if(!on) return;
         
         gun = transform;
         primarySP = gun.Find("PSP");
         rifle = gun.name;
     }
     
     void Start () {
         
     clipsLeft = clips;
     roundsInClip = roundsPerClip;
     }
     
     void Update () {
         
         if(!on) return;
         
         bool hasFiredPrimary = false;
         bool canFire = clipsLeft >= 0;
         reloading = false;
         
         if(clipsLeft <= 0){
             clipsLeft = 0;
         }
         
         if(Input.GetMouseButton(0) && Time.time > nextFire && canFire && roundsInClip > 0){
         nextFire = Time.time + fireRate;
         hasFiredPrimary = true;
         Instantiate(round, primarySP.position, primarySP.rotation);
             //animation.Play("Firing");
             //audio.PlayOneShot(shot);
     }
     
           if(hasFiredPrimary){
               roundsInClip--;
           }
         
         if(roundsInClip == 0 && clipsLeft > 0){
             reloading = true;
             Invoke("ReloadAmmo", 3);
             //animation.Play("Reload");
         }
     }
     
     void ReloadAmmo () {
         
     roundsInClip = roundsPerClip;
     if(reloading){
     clipsLeft --;
     }
     }
     
     void OnTriggerEnter (Collider collider) {
         
         if(collider.CompareTag(rifle)){
             roundsInClip += 15;
             clipsLeft += 0;
         }
     }
 }

I wasn't sure how to implement my strategy, and since everything I tried didn't work, I posted the clean version of the script.

Edit

Okay, I made some progress regarding this. I used transform.Rotate, to rotate to shootpoint, but the problem is that it rotates, but then it doesn't go back to it's original rotation. How should I do that? Here's the script modifications :

 if(Input.GetMouseButton(0) && Time.time > nextFire && canFire){
                 nextFire = Time.time + fireRate;
                 hasFiredPrimary = true;
                 Instantiate(round, primarySP.position, primarySP.rotation);
                 AccuracyModifier();
                 animation.Play("Firing");
                 audio.PlayOneShot(shot);
             }

And later on...

 void AccuracyModifier () {
         
         float x = Random.Range(minAccuracy, maxAccuracy);
         float y = Random.Range(minAccuracy, maxAccuracy);
         float z = Random.Range(minAccuracy, maxAccuracy);
     
         primarySP.Rotate(x,y,z);
     }

I have no idea how to set it back to it's original rotation. I can't set it's original rotation using the Start void because it takes the starting rotation. I know that in-between each shot, I want the rotation to go back the way it was based on how it would be before AccuracyModifier() took place.