Getting colors from ColorArray

Category: kinect for windows v2 sdk

Question

Sergio Bromberg on Mon, 11 Sep 2017 13:35:18


Hi, 

I am using the Unity SDK to build a point cloud from Kinects' data. I have succesfully set the positions of a particle system's particles from depth data but I have trouble extracting color. I am starting from the CoordinateMapperManager.cs file.

This is my code:

void ProcessFrame() { var pDepthData = GCHandle.Alloc(pDepthBuffer, GCHandleType.Pinned); var pDepthCoordinatesData = GCHandle.Alloc(m_pDepthCoordinates, GCHandleType.Pinned); var pColorData = GCHandle.Alloc(pColorBuffer, GCHandleType.Pinned); // Vertex and color lists List <Vector3> vertexList = new List<Vector3>(); List <Color32> colorList = new List<Color32>(); for (int i = 0; i < cDepthWidth; i++) { for(int j = 0 ; j < cDepthHeight ; j++){ int body_index = j*cDepthWidth + i; // If theres a user in the pixel if( pBodyIndexBuffer[body_index] != 0xff){ int depth = pDepthBuffer[ body_index]; // Depth information Vector3 vertex = new Vector3( depth * depthLookupTableVec2[body_index].x , depth * depthLookupTableVec2[body_index].y , depth) ; vertexList.Add ( vertex ); // Color information DepthSpacePoint depthPoint; depthPoint.X = i; depthPoint.Y = j; ColorSpacePoint colorPoint = m_pCoordinateMapper.MapDepthPointToColorSpace(depthPoint, (ushort)depth); int color_index_x = (int) (colorPoint.X + 0.5f); int color_index_y = (int) (colorPoint.Y + 0.5f); byte r = pColorBuffer[color_index_y*cColorWidth + color_index_x]; byte g = pColorBuffer[color_index_y*cColorWidth + color_index_x + 1]; byte b = pColorBuffer[color_index_y*cColorWidth + color_index_x + 2]; byte a = pColorBuffer[color_index_y*cColorWidth + color_index_x + 3]; Color32 color = new Color32(r, g, b, a); colorList.Add(color); } } } //print(sample_byte); // Particle system ParticleSystem.Particle[] particles = new ParticleSystem.Particle[ vertexList.Count]; myParticleSystem.GetParticles(particles); for(int i = 0 ; i < vertexList.Count ; i++){ particles[i].position = vertexList[i]* 0.01f; particles[i].startColor = colorList[i]; setDefaultsForParticle(particles[i]); } if (getSnapshot == true) { // Grab 30 values for(int i = 0; i < 30 ; i++){ int particleIndex = UnityEngine.Random.Range(0, particles.Length); print ( particles[particleIndex].position ); print( "Colors"); print(particles[i].startColor.r); print(particles[i].startColor.g); print(particles[i].startColor.b); print(particles[i].startColor.a); } getSnapshot = false; } myParticleSystem.SetParticles (particles, particles.Length); myParticleSystem.Emit (particles.Length);

}


I wonder if I am sampling incorrectly the color image. I am obtaining red, green and blue lines of color:

https://imgur .com/a/e3Hy9

Any clues on how to solve this?


Replies

Nikolaos Patsiouras on Mon, 11 Sep 2017 14:16:37


First off, why are you pinning memory inside ProcessFrame? You want to do it on Awake/Start , not on a per-frame call.

Secondly, when copying over the new frame's color to the buffer, are you doing a CopyRaw... or a CopyConverted... call? I mean, are you sure the data copied over from Kinect are converted to an RGBA format?

Sergio Bromberg on Mon, 11 Sep 2017 15:24:39


Thank you Nikolau.

1. While I do understand that the 

var pDepthData = GCHandle.Alloc(pDepthBuffer, GCHandleType.Pinned);

line avoids the garbage collector to collect pDepthBuffer, it is not clear for me why this should be done or not in a per-frame basis. I kept it here because this is how it is done in the examples. I just added the line for the color data.

2. I am indeed converting to RGBA format in the update function:

var pColorData = GCHandle.Alloc (pColorBuffer, GCHandleType.Pinned);
pColorFrame.CopyConvertedFrameDataToIntPtr(pColorData.AddrOfPinnedObject(), (uint)pColorBuffer.Length, ColorImageFormat.Rgba);
pColorData.Free();

I wonder if my lines for accessing the color data are correct...

Nikolaos Patsiouras on Mon, 11 Sep 2017 18:31:44


1) Since C# is managed, everything can be collected by the GC. Every once in a while it collects stuff. To avoid data being collected without your permission you pin memory, meaning you notify the GC that this chunk of memory is not to be touched until further notice. The ID that describes that memory is the GCHandle. Ideally you want to pin memory once on initialization(In Unity Awake/Start) and free it once on destroy(OnDestroy/OnApplicationQuit).

I asked because ProcessFrame is probably a function that is being called in either Update or LateUpdate. Even in the samples, it's a function that is called at least one per frame. So what you're doing is spamming the GC about memory being pinned and being freed a lot. So it doesn't make much sense. And there is the hypothetical scenario that the GC does collect the data and incurs a spike because it just so happened in between Free and Alloc.

Always Free as many as you Alloc!!!

2)CopyConvertedFrame seems ok to me.

Have you tried dumping the buffer in a texture and checking the image?

Sergio Bromberg on Tue, 12 Sep 2017 04:37:20


Thank you Nikolaos!

I'll correct the GCHandle line.

I tried dumping the color array in a texture and it is correct. I wonder if the problem lies in how i am accesing the pixels:



byte r = pColorBuffer[color_index_y*cColorWidth + color_index_x];
byte g = pColorBuffer[color_index_y*cColorWidth + color_index_x + 1];
byte b = pColorBuffer[color_index_y*cColorWidth + color_index_x + 2];
byte a = pColorBuffer[color_index_y*cColorWidth + color_index_x + 3];

Color32 color = new Color32(r, g, b, a);


Nikolaos Patsiouras on Tue, 12 Sep 2017 08:22:34


Actually they are.These calculations always confuse me for some time(1D access of a 2D resource).

Your pColorBuffer is of a type that has a size of 4bytes right? Like an int[] or a float[]. If so, then accessing it using pixel coordinates(color_index_..) would give you an int or a float back, the whole color's worth of bytes.

You will have to convert it to bytes with something like this, or perhaps careful use of byte operations.

If pColorBuffer is a byte array though then , I think it's color_y*cColorWidth*cColorBytePerPixel +color_x*cColorBytePerPixel ,where cColorBytePerPixel is 4,to access the red channel and +X for g,b,a. You only calculated the pixel part but forgot to take into account the size in bytes of the pixel for all those bytes you're passing over.


Sergio Bromberg on Tue, 12 Sep 2017 14:44:15


Can't believe I made such a silly mistake!

Thanks Nikolaos! I am able to get correct colors now. Quite surprinsingly, when I query the number of bytes per pixel of the color frame, the output is 2. Is this a mistake from the library?

pColorFrame.FrameDescription.BytesPerPixel;


Nikolaos Patsiouras on Tue, 12 Sep 2017 15:50:58


Not a mistake. I asked you if you used CopyConverted to see if you knew about YUV2.

When you grab the immediate frame description of the Color source, it describes YUV2 which is half of RGBA , 2bytes.

The raw data you get from the sensor are in YUV2 format due to its size. Faster to get through from the sensor to the Kinect service.

Then you convert it to RGBA and do whatever you want.

To get a proper frame description about the RGBA format you have to call CreateFrameDescription and give it the format you wish to query for. It will return the description for it.


UPDATE: The samples query the color format of the source first and act accordingly. But they query for Bgra first. Supposedly there have been sensors that used a different source format so you have to query first and see whether the source outputs the format you want, in case you can call CopyRaw.. ,which is faster than CopyConverted since there's no conversion involved.

Sergio Bromberg on Fri, 03 Nov 2017 04:04:39


Nikolaus,

Sorry to jump into this thread again. I'm trying to clean the code as much as I can. The update function of the samples uses several allocations and deallocations, both in Update() and later in ProcessFrame().


	
	void Update()
	{
		// Get FPS
		elapsedCounter+=Time.deltaTime;
		if(elapsedCounter > 1.0)
		{
			fps = frameCount / elapsedCounter;
			frameCount = 0;
			elapsedCounter = 0.0;
		}

		if (m_pMultiSourceFrameReader == null) 
		{
			return;
		}

		var pMultiSourceFrame = m_pMultiSourceFrameReader.AcquireLatestFrame();
		if (pMultiSourceFrame != null) 
		{
			frameCount++;
			nullFrame = false;

			using(var pDepthFrame = pMultiSourceFrame.DepthFrameReference.AcquireFrame())
			{
				using(var pColorFrame = pMultiSourceFrame.ColorFrameReference.AcquireFrame())
				{
					using(var pBodyIndexFrame = pMultiSourceFrame.BodyIndexFrameReference.AcquireFrame())
					{
						// Get Depth Frame Data.
						if (pDepthFrame != null)
						{
							var pDepthData = GCHandle.Alloc (pDepthBuffer, GCHandleType.Pinned);
							pDepthFrame.CopyFrameDataToIntPtr(pDepthData.AddrOfPinnedObject(), (uint)pDepthBuffer.Length * sizeof(ushort));
							pDepthData.Free();
						}
						
						// Get Color Frame Data
						if (pColorFrame != null)
						{
							var pColorData = GCHandle.Alloc (pColorBuffer, GCHandleType.Pinned);
							pColorFrame.CopyConvertedFrameDataToIntPtr(pColorData.AddrOfPinnedObject(), (uint)pColorBuffer.Length, ColorImageFormat.Rgba);
                            pColorData.Free();
                        }
                        
                        // Get BodyIndex Frame Data.
                        if (pBodyIndexFrame != null)
                        {
							var pBodyIndexData = GCHandle.Alloc (pBodyIndexBuffer, GCHandleType.Pinned);
							pBodyIndexFrame.CopyFrameDataToIntPtr(pBodyIndexData.AddrOfPinnedObject(), (uint)pBodyIndexBuffer.Length);
							pBodyIndexData.Free();
                        }


					}
				}
			}

			ProcessFrame();
        }
        else
		{
			nullFrame = true;
		}
	}

	void ProcessFrame()
	{
		var pDepthData = GCHandle.Alloc(pDepthBuffer, GCHandleType.Pinned);
		var pDepthCoordinatesData = GCHandle.Alloc(m_pDepthCoordinates, GCHandleType.Pinned);

		m_pCoordinateMapper.MapColorFrameToDepthSpaceUsingIntPtr(
			pDepthData.AddrOfPinnedObject(), 
			(uint)pDepthBuffer.Length * sizeof(ushort),
			pDepthCoordinatesData.AddrOfPinnedObject(), 
			(uint)m_pDepthCoordinates.Length);

		pDepthCoordinatesData.Free();
		pDepthData.Free();

		m_pColorRGBX.LoadRawTextureData(pColorBuffer);
		m_pColorRGBX.Apply ();
	}
	
	

So, just to be sure, I should declare the handles in the global scope, allocate in Start() or Awake(), and free in OnApplicationQuit(). Am I right? (and get rid of the allocations and deallocations in frame and process frame)

Something like this:

	GCHandle pDepthData;
	GCHandle pDepthCoordinatesData;
	GCHandle pColorData;
	GCHandle pBodyIndexData;

	void Awake ()
	{...
                pDepthData = GCHandle.Alloc (pDepthBuffer, GCHandleType.Pinned);
		pDepthCoordinatesData = GCHandle.Alloc (m_pDepthCoordinates, GCHandleType.Pinned);
		pColorData = GCHandle.Alloc (pColorBuffer, GCHandleType.Pinned);
		pBodyIndexData = GCHandle.Alloc (pBodyIndexBuffer, GCHandleType.Pinned);

}
...


	void OnApplicationQuit ()
	{	

		pDepthCoordinatesData.Free ();
		pDepthData.Free ();
		pColorData.Free ();
		pBodyIndexData.Free ();
}



Nikolaos Patsiouras on Fri, 03 Nov 2017 08:07:54


Yap. You should be able to see the result in the Profiler Window. This script should be 0 bytes most of the time(Unless you do something else you can't avoid allocating for).