Gaps in C# FeaturePoint Enum - Names Cover 71 of the 121 Feature Points

Category: kinect for windows v1 sdk

Question

kengr on Wed, 12 Sep 2012 21:12:06


There appear to be 121 feature points returned from either FaceTrackFrame.Get3DShape or FaceTrackFrame.GetProjected3DShape, but the FeaturePoint enum in Utils.cs defines names for only 71 of them.  Is there a symbolic definition somewhere for the remaining 50 points? 

I can look at each of the unnamed points and create my own names for them, but before going through that process, I wanted to find out if standard names already exist for them. 

The set of 206 triangles returned by FaceTrackFrame.GetTriangles looks like it refers to all 121 feature points.  I wanted to look at which feature points were involved in each triangle using the following approach after getting the triangles from the frame:

index = 0;

foreach (FaceTriangle triangle in faceTriangles)
{
    Debug.WriteLine(string.Format("{3}: ({0}, {1}, {2})", 
                                  (FeaturePoint)(triangle.First), 
                                  (FeaturePoint)(triangle.Second), 
                                  (FeaturePoint)(triangle.Third), index++));
}

For the feature points with defined enum symbols, the output is helpful. But for the feature points that aren't included in the enum, there's some additional homework required:

...
33: (RightOfLeftEyebrow, MiddleTopOfLeftEyebrow, MiddleBottomOfLeftEyebrow)
34: (MiddleTopOfLeftEyebrow, LeftOfLeftEyebrow, MiddleBottomOfLeftEyebrow)
35: (MiddleBottomOfLeftEyebrow, LeftOfLeftEyebrow, AboveMidUpperLeftEyelid)
36: (OuterCornerOfLeftEye, LeftOfLeftEyebrow, 62)
37: (LeftOfLeftEyebrow, 47, 62)
38: (28, 29, 27)
39: (29, OuterCornerOfRightEye, 27)
40: (26, UnderMidBottomRightEyelid, 25)
41: (58, UnderMidBottomLeftEyelid, 59)
42: (60, OuterCornerOfLeftEye, 62)
43: (60, 62, 61)
44: (26, 111, RightTopDipUpperLip)
45: (26, 75, 111)
46: (25, 75, 26)
47: (59, 76, 58)
48: (112, 76, 59)
49: (LeftTopDipUpperLip, 112, 59)
50: (RightTopDipUpperLip, 6, MiddleTopDipUpperLip)
...

Is there a master list of names for the unnamed feature points somewhere?  Has someone already done this for the next SDK release?  Thanks.

Replies

wmdigital on Wed, 14 Nov 2012 17:54:51


The definitive list appears to be within the Microsoft.Kinect.Toolkit.FaceTracking.FeaturePoint enum itself. In VS use "Go To Definition" or F12 to see all the defined values.

I had the same question and after scouring forums and various search engines there isn't another really good source.

Hope that helps, 
liam m-

kengr on Wed, 14 Nov 2012 19:30:16


As I said in the original posting, the FeaturePoint enum defines only 71 of the 121 points tracked (as illustrated by the code snippet).  What I'm looking for is whether there is already a definition somewhere of the remaining 50 points that are not listed in the FeaturePoint enum.  I would define them myself, but don't want to write application code against a different set of names, if those remaining 50 points are symbolically defined somewhere else already.

Update: This remains an issue in SDK 1.7.  71 of the 121 tracked points are defined in the FeaturePoint enum, but 50 points remain without symbolic definitions.

virgula24 on Mon, 22 Apr 2013 04:42:24


Hi kengr!

Im having the same issues you have. Have you tried plotting the points kinect gives with matlab? i did it and it gives a confusing drawing, seems like a face but it isn't. 

I have a emotion classifier that uses another set of coordinates and I'm trying to find a scale and map one another, but kinect is difficulting my work because the points of the asian guy photo in  the documentation are not the same on the code it self. You can check my pics here: 
dropbox com/sh/e48urjftwsb77zr/74MHd9R2fj

Even if i limit the points to only 87, the drawing still isn't the same as the asian guy (points above eyebrows, middle of cheeks,etc..). Oh and i tried all the methods that return points on the managed code. 

Do you have any thoughts on this?

Thanks

Btw you can check the other system i use, porto.bmp, i was expecting those 87  kinect points would be something like that.

kengr on Mon, 22 Apr 2013 16:07:56


I don't pay any attention at all to the 87-point illustration in the documentation, because it's so misleading.  If you plot the 206 triangles returned by FaceTrackFrame.GetTriangles, all of the 121 points that make up the triangles seem to be positioned well on the face.  If you use the bit of code in my orignal post on this thread, you can see a list of which specific points (either by name, if any, or by index) make up each triangle.  I would suggest using this as a starting point (pun intended) to figure out which specific points (or triangles) you're really interested in.  I would suggest looking at each triangle in turn, to see which triangles contain the parts of the face you want to focus on, and then look at only those points that make up those triangles.

I don't have any issue with the set of 121 points that are returned...they seem to be positioned reasonably well on the face, when viewed in the context of the face triangles.  My only issue is that 50 of the 121 returned points remain without any symbolic names; 71 of the points are named in the FeaturePoint enum, but 50 remain unnamed.

virgula24 on Wed, 24 Apr 2013 02:55:03


hi kengr. I did and experiment and it worked. i was trying to create a new image like the asian guy for the community, but the kinect gives so many duplicate points that i quit. I did however , create a graphic with all the points mapped from the array, so that you and others can consult it. I did it in matlab, i provide the points, the script, and the graphic files (jpg and fig files)in the following link. hope it will be usefull for you and others.

dropbox.com/sh/n2u43r5omfcl3b6/WYmKpOA5Wu

i forgot to mention that it starts in 1 instead of 0, but it can be easily fixed in the script.

Gawdspit on Mon, 29 Apr 2013 10:15:29


Great work, virgula24!

i dont know much about matlab but im glad the points actually falls in place nicely. That brings me to the question of the purpose of the list 71 points that has been in Utils.cs since the beginning of time. What purpose does it serve? It doesn't seem like to be useful for any purpose except for misleading us.

Could you please rotate the matlab plots in such a manner that allows us to clearly see all the number label of each point on the face? Nevermind about creating another misleading asian guy diagram, the triangle mesh works just fine.

Btw, I was googling the contents of the list (just a few random lines like OuterBottomLeftPupil = 70, InnerTopRightPupil = 71, InnerBottomRightPupil = 72,) and found this diagram which seems pretty legit, but the point numbers were labeled differently. Could it be of some use?

http://www.nuicapture.com/img/face_model_diagram.pdf

EDIT: I got a friend to run the matlab scripts for me and I realized that the triangles are defined by yourself in "mesh triangle points.txt". The question is, how did you get to know which 3 points should be linked up to make a triangle? Also,  could you change the color of the plot as the darker blue areas makes it hard to see the point labels.

Alby87 on Mon, 27 May 2013 11:56:46


Hi! I'm working on a program using the Face Tracking Interface. I'm needing points that aren't on Utilis.cs, the asian guy diagram is known to be wrong, and the Nuicapture diagram is also wrong, and I'll prove that.

I need the distance between the eyelid and the "above eyelid". In utils.cs "Above Eyelid" is not fixed, so I used the Nuicapture diagram, and I calculated points 53 and 55

53.  AboveMidUpperRightEyelid
55.  MiddleTopRightEyelid

and drawed those points. The bad thing is that these points are wrong. I searched in utils.cs, and I found

OuterCornerOfLeftEye            = 53,

MiddleBottomLeftEyelid          = 55,

that's are the points I found with a text program, blinking those pixel over my face! So utils.cs is correct, and NUI capture is "wrong" . So, let's try the virgula24 plotted diagram... and it says that the points I need are 53 and 55! So they are also wrong...

I'm using the updated SDK (1.7) and Toolkit... There is a complete, definite, correct point graph? Many thanks :D

Alby87

P.s. Obviously, I noticed that between NUICapture diagram and Virgula diagram all points are MIRRORED, but this is not the problem:

I noticed that in NUI diagram all points are added to 1! So, if you use the points in the diagram, remeber to mirror them and subtract 1!

Carmine Si - MSFT on Thu, 30 May 2013 00:53:13


The list of points are based on vertices in the face mesh. FaceTracking uses a mesh of a fixed vertex count that is morphed to the face. Only the points that have a significance(corner of the mouth) are named. The other points will just be vertices that make up the rest of the mesh model.

kengr on Sat, 01 Jun 2013 17:50:02


Carmine -

Thanks for your post. 

Are you saying that the 50 missing points from the FeaturePoint enum have been left undefined by design, even though they are used as vertices in many of the 206 face triangles returned by FaceTrackFrame.GetTriangles?  I just want to understand if it was a conscious decision to leave out the 50 point names.

Carmine Si - MSFT on Mon, 03 Jun 2013 21:33:46


Yes, this was by-design. I can take any of your feedback and relay this back to the team. If you have any particular scenarios where this may be an oversight, we can certainly try to address that.

kengr on Mon, 03 Jun 2013 23:08:26


Carmine -

Thanks for your definitive answer.

In an ideal world, I'd like to have symbolic names for all of the remaining 50 points, to aid in debugging and in selecting subsets of triangles to examine or display.  This would be preferable to working with a mix of symbols and hard-coded integer values (as shown in my original post) when looking at face triangle data.  Given that the absence of these 50 point symbols is by design, I will probably go ahead and create my own to use for this purpose.  I was hoping to avoid this, if there had been plans to add the remaining symbols in a future SDK release.

Thanks again.

OuGhss on Thu, 29 May 2014 17:54:30


I see that the diagram seems legit, but if someone from developers of the sdk could telle us if it represents the correct number for each feature point or not.

Andressaa Leite on Tue, 23 Jan 2018 18:33:32


did you can the list?