Wayback Machinekoobas.hobune.stream
May JUN Jul
Previous capture 13 Next capture
2021 2022 2023
1 capture
13 Jun 22 - 13 Jun 22
sparklines
Close Help
  • Products
  • Solutions
  • Made with Unity
  • Learning
  • Support & Services
  • Community
  • Asset Store
  • Get Unity

UNITY ACCOUNT

You need a Unity Account to shop in the Online and Asset Stores, participate in the Unity Community and manage your license portfolio. Login Create account
  • Blog
  • Forums
  • Answers
  • Evangelists
  • User Groups
  • Beta Program
  • Advisory Panel

Navigation

  • Home
  • Products
  • Solutions
  • Made with Unity
  • Learning
  • Support & Services
  • Community
    • Blog
    • Forums
    • Answers
    • Evangelists
    • User Groups
    • Beta Program
    • Advisory Panel

Unity account

You need a Unity Account to shop in the Online and Asset Stores, participate in the Unity Community and manage your license portfolio. Login Create account

Language

  • Chinese
  • Spanish
  • Japanese
  • Korean
  • Portuguese
  • Ask a question
  • Spaces
    • Default
    • Help Room
    • META
    • Moderators
    • Topics
    • Questions
    • Users
    • Badges
  • Home /
avatar image
2
Question by joshrs926 · May 27, 2021 at 01:31 AM · listresizedotsnativeunsafe

How to implement custom Native List

I am trying to write an editor extension asset that I hope to post to the asset store. For this reason I don’t want it to depend on any preview packages such as the preview Unity.Collections package which contains NativeList (not to be confused with Unity.Collections which is in the core engine by default). But I want to be able to use NativeList functionality. Specifically, I want to be able to add elements to and resize a native collection within a Bursted IJob, which NativeList is able to do. How could I achieve this? I am happy to use unsafe code such as UnsafeUtility.Malloc but I’m not familiar with it. Some example code would be very helpful.

Comment
Add comment · Show 6
10 |3000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image andrew-lukasik · May 27, 2021 at 07:07 AM 0
Share

You're saying that you're happy to use Unity.Collections.LowLevel.Unsafe.UnsafeUtility.Malloc() so what is stopping you from doing just that?

avatar image joshrs926 andrew-lukasik · May 27, 2021 at 11:57 AM 0
Share

Am I able to Malloc with a TempJob allocator from within a job to allocate a new double sized array and copy the old array into the new one? This is what a list does when you add elements beyond its capacity. But I didn’t think you could use TempJob or Persistent allocators from within a job. Somehow though NativeList is able to expand its capacity from within a job and have that change persist outside the job.

avatar image andrew-lukasik joshrs926 · Jun 06, 2021 at 07:04 PM 0
Share

But I didn’t think you could use TempJob or Persistent allocators from within a job.

You can. Safe containers just prevent us from doing that.

Show more comments

1 Reply

· Add your reply
  • Sort: 
avatar image
3

Answer by andrew-lukasik · May 27, 2021 at 02:28 PM

Safe option

Have you considered creating a simple fixed-size list instead? Those could be most compatible, as one can build them out of NativeArrays.

Consider something like this:

( just an untested sketch )

 using Unity.Collections;
 public struct NativeListFixedSize <T> : System.IDisposable
     where T : unmanaged
 {
     NativeArray<T> _data;
     NativeArray<int> _meta;
     public int Capacity => _data.Length;
     int _index { get => _meta[0]; set => _meta[0]=value; }
     public int Length => _index;
     public NativeListFixedSize ( int capacity  , Allocator allocator )
     {
         this._data = new NativeArray<T>( capacity , allocator );
         this._meta = new NativeArray<int>( 1 , allocator );
         this.Clear();
     }
     public bool Add ( T value )
     {
         if( _index < Capacity ) { _data[_index++] = value; return true; }
         else return false;
     }
     /// <remarks> last element in list moves to given position, to fill the gap </remarks>
     public bool RemoveAtFast ( int i )
     {
         if( i<0 || i>=Length ) return false;
         _data[i] = _data[Length-1];
         _index--;
         return true;
     }
     public void Clear () => this._index = 0;
     public NativeSlice<T> Slice () => this._data.Slice( 0 , this.Length );
     public void Dispose ()
     {
         this._data.Dispose();
         this._meta.Dispose();
     }
     public void GetDataAndDispose ( out NativeArray<T> data )
     {
         data = this._data;
         this._meta.Dispose();
     }
 }

side note: Slice() is ideal for iteration and general access without exposing internal fields

 var slice = nativeListFixedSize.Slice();
 for( int i=0 ; i<slice.Length ; i++ )
     slice[i] += 1;

In theory, list can be infinite, but we know it's not true at all. And this observation can make us think about it more deeply and decide list capacity up front. If that's done - it's just a matter of increasing/decreasing an allocated index when adding/removing a value.

But for those cases where list capacity may be an issue you can test:

 bool wasCapacityReached = nativeListFixedSize.Length==nativeListFixedSize.Capacity;

and address it by, for example, doubling list's capacity and running the job again.


Unsafe option

Word of friendly warning first: pointers can (i.e. will) cause application crashing to desktop, editor included, without even a single warning.

VeryUnsafeList.cs

 // src*: https://gist.github.com/andrew-raphael-lukasik/09c8a9c29bb5548ea65273653474f8f1
 using UnityEngine;
 using UnityEngine.Assertions;
 using Unity.Collections;
 using Unity.Collections.LowLevel.Unsafe;
 
 public unsafe struct VeryUnsafeList <T> : System.IDisposable where T : unmanaged
 {
     public T* ptr;
     public readonly Allocator allocator;
     public int length;
     public int capacity;
     public int size { get; private set; }
     public VeryUnsafeList ( Allocator allocator )
         : this( capacity:0 , allocator:allocator ) {}
     public VeryUnsafeList ( int capacity , Allocator allocator )
     {
         Assert.IsFalse( capacity<0 , "invalid capacity" );
         Assert.IsFalse( allocator==Allocator.Invalid , "invalid allocator" );
 
         this.length = 0;
         this.capacity = capacity;
         this.allocator = allocator;
 
         this.ptr = null;
         this.size = 0;
         this.Resize( newCapacity:capacity );
     }
     public T this [ int index ]
     {
         get
         {
             if( index<0 || index>=this.length ) throw new System.IndexOutOfRangeException();
             return this.ptr[index];
         }
         set
         {
             if( index<0 || index>=this.length ) throw new System.IndexOutOfRangeException();
             this.ptr[index] = value;
         }
     }
     public void Add ( T value )
     {
         if( this.capacity==0 )
         {
             this.Resize( 1 );
             Debug.Log($"\tresized from {0} to {this.capacity} ({this.size} bytes)");
         }
         if( this.length==this.capacity )
         {
             int old = this.capacity;
             this.Resize( this.capacity * 2 );
             Debug.Log($"\tresized from {old} to {this.capacity} ({this.size} bytes)");
         }
         this.ptr[this.length++] = value;
     }
     public void Remove ( T value )
     {
         if( this.length==0 ) return;
         for( int i=0 ; i<this.length ; i++ )
         if( this.ptr[i].Equals(value) )
             this.ptr[i] = this.ptr[--this.length];
     }
     public void RemoveAt ( int index )
     {
         if( index<0 || index>=this.length ) throw new System.IndexOutOfRangeException();
         this.ptr[index] = this.ptr[--this.length];
     }
     public void Clear () => this.length = 0;
     public void Resize ( int newCapacity )
     {
         int newSize = sizeof(T) * newCapacity;
         T* newPtr = (T*) UnsafeUtility.Malloc( size:newSize , alignment:4 , allocator:this.allocator );
         if( this.ptr!=null )
         {
             UnsafeUtility.MemCpy( destination:newPtr , source:this.ptr , size:Mathf.Min(this.size,newSize) );
             this.Dispose();
         }
         this.capacity = newCapacity;
         this.ptr = newPtr;
         this.size = newSize;
     }
     public void Dispose ()
     {
         if( this.ptr!=null )
         {
             UnsafeUtility.Free( this.ptr , this.allocator );
             this.ptr = null;
             this.size = 0;
         }
     }
     public override string ToString () => $"{{ {nameof(ptr)}:{(long)ptr} , {nameof(allocator)}:{allocator} , {nameof(length)}:{length} , {nameof(capacity)}:{capacity} , {(nameof(size))}:{size} }}";
 }
 

AdventuresInUnsafeAllocations.cs

 using UnityEngine;
 using Unity.Collections;
 using Unity.Collections.LowLevel.Unsafe;
 using Unity.Jobs;
 
 using NaughtyAttributes;
 
 public unsafe class AdventuresInUnsafeAllocations : MonoBehaviour
 {
     [SerializeField] Allocator mainThreadAllocator = Allocator.Persistent;
     [SerializeField] Allocator jobAllocator = Allocator.Temp;
 
     void OnEnable () => Run();
 
     [Button("GO")]
     void Run ()
     {
         VeryUnsafeList<int>* dataPtr = (VeryUnsafeList<int>*) UnsafeUtility.Malloc( size:sizeof(VeryUnsafeList<int>) , alignment:4 , allocator:mainThreadAllocator );
         *dataPtr = new VeryUnsafeList<int>( jobAllocator );
         
         Debug.Log($"run started");
         var job = new AllocationsJob{ dataPtr = dataPtr };
         job.Schedule().Complete();
 
         var data = *dataPtr;
         var text = new System.Text.StringBuilder();
         for( int i=0 ; i<data.length ; i++ )
             text.Append( data.ptr[i] ).Append( i<data.length-1 ? ',' : ' ' );
         Debug.Log($"({data.length}) output data: {{ {text} }}");
         
         data.Dispose();
         UnsafeUtility.Free( dataPtr , mainThreadAllocator );
         Debug.Log($"run completed using {data.allocator} job allocator");
     }
 }
 
 public unsafe struct AllocationsJob : IJob
 {
     [NativeDisableUnsafePtrRestriction] public VeryUnsafeList<int>* dataPtr;
     public void Execute ()
     {
         (*dataPtr).Add( 999 );
         for( int i=0 ; i<10 ; i++ )
             (*dataPtr).Add( i );
         (*dataPtr).Add( 11 );
         (*dataPtr).Add( -1 );
         (*dataPtr)[11] = (*dataPtr)[12];
         (*dataPtr).Remove( 999 );
         (*dataPtr).RemoveAt( 11 );
     }
 }
Comment
Add comment · Show 3 · Share
10 |3000 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users
avatar image joshrs926 · May 27, 2021 at 03:09 PM 0
Share

Thanks a lot for your reply Andrew. This is basically what I’m doing now but my goal is to have a container that will allocate a new array of double size and copy elements into it from inside a job and have that new array persist outside the job and be used for future jobs. It seems that NativeList and UnsafeList are able to do this, but I don’t want to use Preview Packages in my asset. Also the jobs are “lazy jobs” meaning they can last multiple frames.

avatar image andrew-lukasik joshrs926 · Jun 06, 2021 at 06:54 PM 0
Share

I updated my answer. Added a working example of a skeletal unsafe list built on generic pointers T* that can resize mid job execution using Allocator.Persistent.

It lacks any safety facilities though, so don't be surprised by application crashing (yup) to desktop after modifying it. Pointers aren't made for safety, exactly, so consider yourself warned.

avatar image andrew-lukasik · Jul 21, 2021 at 04:16 PM 0
Share

(bugfix)

 UnsafeUtility.MemCpy( destination:newPtr , source:this.ptr , size:this.size );

replaced with:

 UnsafeUtility.MemCpy( destination:newPtr , source:this.ptr , size:Mathf.Min(this.size,newSize) );

Original line would copy too much bytes (classic buffer overflow) when downsizing allocations, which was super bad, because it would corrupt random memory causing crashes in very different parts of the engine for no immediately apparent reason.

See? unsafe keyword is no hyperbole :)

Your answer

Hint: You can notify a user about this post by typing @username

Up to 2 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

Follow this Question

Answers Answers and Comments

124 People are following this question.

avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image avatar image

Related Questions

A node in a childnode? 1 Answer

NativeList Empty Outside Job But Not Inside It 1 Answer

DOTS NativeContainers error 1 Answer

Remove and resize List 1 Answer

Javascript Endless 3D Array 0 Answers


Enterprise
Social Q&A

Social
Subscribe on YouTube social-youtube Follow on LinkedIn social-linkedin Follow on Twitter social-twitter Follow on Facebook social-facebook Follow on Instagram social-instagram

Footer

  • Purchase
    • Products
    • Subscription
    • Asset Store
    • Unity Gear
    • Resellers
  • Education
    • Students
    • Educators
    • Certification
    • Learn
    • Center of Excellence
  • Download
    • Unity
    • Beta Program
  • Unity Labs
    • Labs
    • Publications
  • Resources
    • Learn platform
    • Community
    • Documentation
    • Unity QA
    • FAQ
    • Services Status
    • Connect
  • About Unity
    • About Us
    • Blog
    • Events
    • Careers
    • Contact
    • Press
    • Partners
    • Affiliates
    • Security
Copyright © 2020 Unity Technologies
  • Legal
  • Privacy Policy
  • Cookies
  • Do Not Sell My Personal Information
  • Cookies Settings
"Unity", Unity logos, and other Unity trademarks are trademarks or registered trademarks of Unity Technologies or its affiliates in the U.S. and elsewhere (more info here). Other names or brands are trademarks of their respective owners.
  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Spaces
  • Default
  • Help Room
  • META
  • Moderators
  • Explore
  • Topics
  • Questions
  • Users
  • Badges