- Home /
How to implement custom Native List
I am trying to write an editor extension asset that I hope to post to the asset store. For this reason I don’t want it to depend on any preview packages such as the preview Unity.Collections
package which contains NativeList
(not to be confused with Unity.Collections which is in the core engine by default). But I want to be able to use NativeList
functionality. Specifically, I want to be able to add elements to and resize a native collection within a Burst
ed IJob
, which NativeList
is able to do. How could I achieve this? I am happy to use unsafe code such as UnsafeUtility.Malloc
but I’m not familiar with it. Some example code would be very helpful.
You're saying that you're happy to use Unity.Collections.LowLevel.Unsafe.UnsafeUtility.Malloc()
so what is stopping you from doing just that?
Am I able to Malloc with a TempJob allocator from within a job to allocate a new double sized array and copy the old array into the new one? This is what a list does when you add elements beyond its capacity. But I didn’t think you could use TempJob or Persistent allocators from within a job. Somehow though NativeList is able to expand its capacity from within a job and have that change persist outside the job.
But I didn’t think you could use TempJob or Persistent allocators from within a job.
You can. Safe containers just prevent us from doing that.
Answer by andrew-lukasik · May 27, 2021 at 02:28 PM
Safe option
Have you considered creating a simple fixed-size list instead? Those could be most compatible, as one can build them out of NativeArray
s.
Consider something like this:
( just an untested sketch )
using Unity.Collections;
public struct NativeListFixedSize <T> : System.IDisposable
where T : unmanaged
{
NativeArray<T> _data;
NativeArray<int> _meta;
public int Capacity => _data.Length;
int _index { get => _meta[0]; set => _meta[0]=value; }
public int Length => _index;
public NativeListFixedSize ( int capacity , Allocator allocator )
{
this._data = new NativeArray<T>( capacity , allocator );
this._meta = new NativeArray<int>( 1 , allocator );
this.Clear();
}
public bool Add ( T value )
{
if( _index < Capacity ) { _data[_index++] = value; return true; }
else return false;
}
/// <remarks> last element in list moves to given position, to fill the gap </remarks>
public bool RemoveAtFast ( int i )
{
if( i<0 || i>=Length ) return false;
_data[i] = _data[Length-1];
_index--;
return true;
}
public void Clear () => this._index = 0;
public NativeSlice<T> Slice () => this._data.Slice( 0 , this.Length );
public void Dispose ()
{
this._data.Dispose();
this._meta.Dispose();
}
public void GetDataAndDispose ( out NativeArray<T> data )
{
data = this._data;
this._meta.Dispose();
}
}
side note:
Slice()
is ideal for iteration and general access without exposing internal fields
var slice = nativeListFixedSize.Slice();
for( int i=0 ; i<slice.Length ; i++ )
slice[i] += 1;
In theory, list can be infinite, but we know it's not true at all. And this observation can make us think about it more deeply and decide list capacity up front. If that's done - it's just a matter of increasing/decreasing an allocated index when adding/removing a value.
But for those cases where list capacity may be an issue you can test:
bool wasCapacityReached = nativeListFixedSize.Length==nativeListFixedSize.Capacity;
and address it by, for example, doubling list's capacity and running the job again.
Unsafe option
Word of friendly warning first: pointers can (i.e. will) cause application crashing to desktop, editor included, without even a single warning.
VeryUnsafeList.cs
// src*: https://gist.github.com/andrew-raphael-lukasik/09c8a9c29bb5548ea65273653474f8f1
using UnityEngine;
using UnityEngine.Assertions;
using Unity.Collections;
using Unity.Collections.LowLevel.Unsafe;
public unsafe struct VeryUnsafeList <T> : System.IDisposable where T : unmanaged
{
public T* ptr;
public readonly Allocator allocator;
public int length;
public int capacity;
public int size { get; private set; }
public VeryUnsafeList ( Allocator allocator )
: this( capacity:0 , allocator:allocator ) {}
public VeryUnsafeList ( int capacity , Allocator allocator )
{
Assert.IsFalse( capacity<0 , "invalid capacity" );
Assert.IsFalse( allocator==Allocator.Invalid , "invalid allocator" );
this.length = 0;
this.capacity = capacity;
this.allocator = allocator;
this.ptr = null;
this.size = 0;
this.Resize( newCapacity:capacity );
}
public T this [ int index ]
{
get
{
if( index<0 || index>=this.length ) throw new System.IndexOutOfRangeException();
return this.ptr[index];
}
set
{
if( index<0 || index>=this.length ) throw new System.IndexOutOfRangeException();
this.ptr[index] = value;
}
}
public void Add ( T value )
{
if( this.capacity==0 )
{
this.Resize( 1 );
Debug.Log($"\tresized from {0} to {this.capacity} ({this.size} bytes)");
}
if( this.length==this.capacity )
{
int old = this.capacity;
this.Resize( this.capacity * 2 );
Debug.Log($"\tresized from {old} to {this.capacity} ({this.size} bytes)");
}
this.ptr[this.length++] = value;
}
public void Remove ( T value )
{
if( this.length==0 ) return;
for( int i=0 ; i<this.length ; i++ )
if( this.ptr[i].Equals(value) )
this.ptr[i] = this.ptr[--this.length];
}
public void RemoveAt ( int index )
{
if( index<0 || index>=this.length ) throw new System.IndexOutOfRangeException();
this.ptr[index] = this.ptr[--this.length];
}
public void Clear () => this.length = 0;
public void Resize ( int newCapacity )
{
int newSize = sizeof(T) * newCapacity;
T* newPtr = (T*) UnsafeUtility.Malloc( size:newSize , alignment:4 , allocator:this.allocator );
if( this.ptr!=null )
{
UnsafeUtility.MemCpy( destination:newPtr , source:this.ptr , size:Mathf.Min(this.size,newSize) );
this.Dispose();
}
this.capacity = newCapacity;
this.ptr = newPtr;
this.size = newSize;
}
public void Dispose ()
{
if( this.ptr!=null )
{
UnsafeUtility.Free( this.ptr , this.allocator );
this.ptr = null;
this.size = 0;
}
}
public override string ToString () => $"{{ {nameof(ptr)}:{(long)ptr} , {nameof(allocator)}:{allocator} , {nameof(length)}:{length} , {nameof(capacity)}:{capacity} , {(nameof(size))}:{size} }}";
}
AdventuresInUnsafeAllocations.cs
using UnityEngine;
using Unity.Collections;
using Unity.Collections.LowLevel.Unsafe;
using Unity.Jobs;
using NaughtyAttributes;
public unsafe class AdventuresInUnsafeAllocations : MonoBehaviour
{
[SerializeField] Allocator mainThreadAllocator = Allocator.Persistent;
[SerializeField] Allocator jobAllocator = Allocator.Temp;
void OnEnable () => Run();
[Button("GO")]
void Run ()
{
VeryUnsafeList<int>* dataPtr = (VeryUnsafeList<int>*) UnsafeUtility.Malloc( size:sizeof(VeryUnsafeList<int>) , alignment:4 , allocator:mainThreadAllocator );
*dataPtr = new VeryUnsafeList<int>( jobAllocator );
Debug.Log($"run started");
var job = new AllocationsJob{ dataPtr = dataPtr };
job.Schedule().Complete();
var data = *dataPtr;
var text = new System.Text.StringBuilder();
for( int i=0 ; i<data.length ; i++ )
text.Append( data.ptr[i] ).Append( i<data.length-1 ? ',' : ' ' );
Debug.Log($"({data.length}) output data: {{ {text} }}");
data.Dispose();
UnsafeUtility.Free( dataPtr , mainThreadAllocator );
Debug.Log($"run completed using {data.allocator} job allocator");
}
}
public unsafe struct AllocationsJob : IJob
{
[NativeDisableUnsafePtrRestriction] public VeryUnsafeList<int>* dataPtr;
public void Execute ()
{
(*dataPtr).Add( 999 );
for( int i=0 ; i<10 ; i++ )
(*dataPtr).Add( i );
(*dataPtr).Add( 11 );
(*dataPtr).Add( -1 );
(*dataPtr)[11] = (*dataPtr)[12];
(*dataPtr).Remove( 999 );
(*dataPtr).RemoveAt( 11 );
}
}
Thanks a lot for your reply Andrew. This is basically what I’m doing now but my goal is to have a container that will allocate a new array of double size and copy elements into it from inside a job and have that new array persist outside the job and be used for future jobs. It seems that NativeList and UnsafeList are able to do this, but I don’t want to use Preview Packages in my asset. Also the jobs are “lazy jobs” meaning they can last multiple frames.
I updated my answer. Added a working example of a skeletal unsafe
list built on generic pointers T*
that can resize mid job execution using Allocator.Persistent
.
It lacks any safety facilities though, so don't be surprised by application crashing (yup) to desktop after modifying it. Pointers aren't made for safety, exactly, so consider yourself warned.
(bugfix)
UnsafeUtility.MemCpy( destination:newPtr , source:this.ptr , size:this.size );
replaced with:
UnsafeUtility.MemCpy( destination:newPtr , source:this.ptr , size:Mathf.Min(this.size,newSize) );
Original line would copy too much bytes (classic buffer overflow) when downsizing allocations, which was super bad, because it would corrupt random memory causing crashes in very different parts of the engine for no immediately apparent reason.
See? unsafe
keyword is no hyperbole :)
Your answer
Follow this Question
Related Questions
A node in a childnode? 1 Answer
NativeList Empty Outside Job But Not Inside It 1 Answer
DOTS NativeContainers error 1 Answer
Remove and resize List 1 Answer
Javascript Endless 3D Array 0 Answers