Time.time and audio time seem completely inconsistent with each other in the first few frames, difficulty syncing audio and movement

Hey all!

I'm making a game that involves syncing audio and movement up, nothing fancy that needs EXACT precision but it needs to be pretty close, within a 0.1 second or so margin of error.

For the most part it works except... in bigger stages with more going on the audio time (calculated by samples / frequency) can get a second or more off from the movement right at the start. Looking into it, it looks like something weird is going on the first few frames. I'm attaching an image that shows a simple stage on the left (with a slight gap between the Time.time and audio time) and a more complex stage on the right (with a larger gap.)

NO MATTER WHAT, the second frame always returns 0.2 for Time.time and the third frame always returns 0.3533334, and there is no way that can be right, right? So that is probably inaccurate? But why would Unity have inaccurate time for a few frames? Whatever the case, it creates a weird situation like the times on the image on the right below, where in frame 2 Time.time is returning 0.2 and my audio is returning 0, but at frame 3 Time.time has only changed by about 0.15, while my audio time has changed by over 1.0.

I didn't do exact calculations but it seems like after those first few frames it's more or less accurately syncing time.

This is just the worst example I could capture, but at times I've had it off by 2-3 seconds or more from the start.

So like, any insight into what is going on in those first few frames, and why time is being measured differently by my Time.time and audio time? A part of me wants to just start my audio and movement at frame 5 or something and see if that fixes it, but that feels so hacky and I want to actually understand what is going on here before trying to fix it. Because maybe the audio start is causing the problem or something so it wouldn't matter where I moved it. I dunno.

HELP!