Question

clearshot365 on Thu, 09 May 2019 14:02:18


Hi

I have closely followed the example on how to send a NVME Vendor Specific pass-through commands to the device, however i can't figure out why the ERROR_INVALID_FUNCTION code gets returned every single time.

Im working with an NVME device which is developed in house (our own SSD/NVME controller chip) which has been tested and works under linux with no issues.

I have also verified, that the device is able to send the Command Effects Log with all the commands that is supports, here is the output of this log queried under Linux environment:

fpga@host:[~/WORK/nvme-cli/nvme-cli]: sudo ./nvme effects-log /dev/nvme0
Admin Command Set
ACS0     [Delete I/O Submission Queue     ] 00000001
ACS1     [Create I/O Submission Queue     ] 00000001
ACS2     [Get Log Page                    ] 00000001
ACS4     [Delete I/O Completion Queue     ] 00000001
ACS5     [Create I/O Completion Queue     ] 00000001
ACS6     [Identify                        ] 00000001
ACS8     [Abort                           ] 00000001
ACS9     [Set Features                    ] 00000001
ACS10    [Get Features                    ] 00000001
ACS12    [Asynchronous Event Request      ] 00000001
ACS16    [Firmware Commit                 ] 00000001
ACS17    [Firmware Image Download         ] 00000001
ACS128   [Format NVM                      ] 00000001
ACS129   [Security Send                   ] 00000001
ACS130   [Security Receive                ] 00000001
ACS192   [Unknown                         ] 00000001
ACS193   [Unknown                         ] 00000001
ACS194   [Unknown                         ] 00000001
ACS195   [Unknown                         ] 00000001
ACS196   [Unknown                         ] 00000001
ACS197   [Unknown                         ] 00000001
ACS198   [Unknown                         ] 00000001
ACS199   [Unknown                         ] 00000001
ACS200   [Unknown                         ] 00000001

NVM Command Set
IOCS0    [Flush                           ] 00000001
IOCS1    [Write                           ] 00000001
IOCS2    [Read                            ] 00000001
IOCS4    [Write Uncorrectable             ] 00000001
IOCS5    [Compare                         ] 00000001
IOCS8    [Write Zeroes                    ] 00000001
IOCS9    [Dataset Management              ] 00000001
IOCS13   [Reservation Register            ] 00000001
IOCS14   [Reservation Report              ] 00000001
IOCS17   [Reservation Acquire             ] 00000001
IOCS21   [Reservation Release             ] 00000001
IOCS131  [Unknown                         ] 00000001
IOCS132  [Unknown                         ] 00000001
IOCS133  [Unknown                         ] 00000001
IOCS134  [Unknown                         ] 00000001

I have based the test code i have for windows on the example from https://docs.microsoft.com/en-us/windows/desktop/FileIO/working-with-nvme-devices#pass-through-mechanism

I'm able to send a standard IDENTITY command from the same example with no issues and getting a proper response.

However, when i try to send a Vendor Specific Admin command with any command code from 0xC1-0xC6 i  get the error 1 (ERROR_INVALID_FUNCTION)

Here is the code i use to send a Vendor Specific Admin command:

BOOL nvme_vendor_admin_cmd(HANDLE hDevice)
{

	BOOL    result = 0;
	PVOID   buffer = NULL;
	ULONG   bufferLength = 0;
	ULONG   returnedLength = 0;
	PSTORAGE_PROTOCOL_COMMAND protocolCommand = NULL;
	PNVME_COMMAND command = NULL;

	//
	// Allocate buffer for use.
	//

	bufferLength = FIELD_OFFSET(STORAGE_PROTOCOL_COMMAND, Command) +
		STORAGE_PROTOCOL_COMMAND_LENGTH_NVME +  sizeof(NVME_ERROR_INFO_LOG) + NVME_MAX_LOG_SIZE;

	buffer = malloc(bufferLength);

        if(!buffer)
	  return FALSE;
	//
	// Initialize query data structure to get Identify Controller Data.
	//
	ZeroMemory(buffer, bufferLength);

	protocolCommand = (PSTORAGE_PROTOCOL_COMMAND)buffer;
	command = (PNVME_COMMAND)protocolCommand->Command;

	protocolCommand->Version = STORAGE_PROTOCOL_STRUCTURE_VERSION;
	protocolCommand->Length = sizeof(STORAGE_PROTOCOL_COMMAND);
	protocolCommand->ProtocolType = ProtocolTypeNvme;
	protocolCommand->Flags = STORAGE_PROTOCOL_COMMAND_FLAG_ADAPTER_REQUEST;
	protocolCommand->CommandLength = STORAGE_PROTOCOL_COMMAND_LENGTH_NVME;
	protocolCommand->ErrorInfoLength = sizeof(NVME_ERROR_INFO_LOG);
	protocolCommand->DataFromDeviceTransferLength = NVME_MAX_LOG_SIZE;
	protocolCommand->TimeOutValue = 10;
	protocolCommand->ErrorInfoOffset = FIELD_OFFSET(STORAGE_PROTOCOL_COMMAND, Command) + STORAGE_PROTOCOL_COMMAND_LENGTH_NVME;
	protocolCommand->DataToDeviceBufferOffset = protocolCommand->ErrorInfoOffset + protocolCommand->ErrorInfoLength;
	protocolCommand->CommandSpecific = STORAGE_PROTOCOL_SPECIFIC_NVME_ADMIN_COMMAND;

	command->CDW0.OPC = 0xC3; // opcode;
	command->u.GENERAL.CDW10 = 256; 
	command->u.GENERAL.CDW11 = 0;
	command->u.GENERAL.CDW12 = 0;
	command->u.GENERAL.CDW13 = 0;
	command->u.GENERAL.CDW14 = 0;

	//
	// Send request down.
	//
	result = DeviceIoControl(hDevice,
		IOCTL_STORAGE_PROTOCOL_COMMAND,
		buffer,
		bufferLength,
		buffer,
		bufferLength,
		&returnedLength,
		NULL
	);

	if (FALSE == result)
	{
		DWORD dw = GetLastError();
		std::cout << "LastErrorCode: " << dw << "\n";
		goto exit;
	}
	else
	{
		std::cout << "NVME Vendor Admin Command Success: " << dw << "\n";
	}

exit:

	if (buffer)
		free(buffer);

	return result;

}

Please let me know what am i doing wrong since i need to get this to work with our device under windows.

Any help is greatly appreciated



Replies

clearshot365 on Thu, 23 May 2019 16:20:12


Anyone from Microsoft care to comment? its been over 2 weeks and not a single reply :(

Cymon Kilmer [MSFT] on Mon, 10 Jun 2019 19:55:03


Hello,

A dev from the feature team reviewed this and statest that the code looks fine and asked does the vendor specific commands send down has corresponding command effects data from command effects log? If not, storport will block any command to the device as mentioned in https://docs.microsoft.com/en-us/windows/desktop/FileIO/working-with-nvme-devices#pass-through-mechanism

Important:

StoreNVMe.sys and Storeport.sys will block any command to a device if it is not described in the Command Effects Log. 

clearshot365 on Thu, 27 Jun 2019 15:01:13


Ok so it looks like the pass-through started to work however we are facing another slew of issues.

Sending the Vendor Specific commands work only if there is no data to be written to the NVMe device, what i mean by that is that we can send the Vendor Specific commands but as soon as we try to send a command that also required a PRP transfer from host to Device, command fails in 2 different ways.

Here is the example code:

DWORD data_len = 1024; 
DWORD buffer_length = 0;
DWORD data_buffer_offset = 0;
int32_t                      retval = 0;
PSTORAGE_PROTOCOL_COMMAND    p_protocol_cmd = NULL;
PNVME_COMMAND                p_nvme_cmd = NULL;
BOOL                         b_retval = FALSE

buffer_length = FIELD_OFFSET(STORAGE_PROTOCOL_COMMAND, Command) +
                    STORAGE_PROTOCOL_COMMAND_LENGTH_NVME + sizeof(NVME_ERROR_INFO_LOG) + data_len;


p_buffer = VirtualAlloc(NULL, buffer_length, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
  if (!p_buffer)
     return -1;

ZeroMemory(p_buffer, buffer_length);

p_protocol_cmd = (PSTORAGE_PROTOCOL_COMMAND)p_buffer;
    p_nvme_cmd     = (PNVME_COMMAND)p_protocol_cmd->Command;

p_protocol_cmd->Version = STORAGE_PROTOCOL_STRUCTURE_VERSION;
p_protocol_cmd->Length = sizeof(STORAGE_PROTOCOL_COMMAND);
p_protocol_cmd->ProtocolType = ProtocolTypeNvme;
p_protocol_cmd->Flags = STORAGE_PROTOCOL_COMMAND_FLAG_ADAPTER_REQUEST;
p_protocol_cmd->CommandLength = STORAGE_PROTOCOL_COMMAND_LENGTH_NVME;
p_protocol_cmd->ErrorInfoLength = sizeof(NVME_ERROR_INFO_LOG);
p_protocol_cmd->TimeOutValue = 10;
p_protocol_cmd->ErrorInfoOffset = FIELD_OFFSET(STORAGE_PROTOCOL_COMMAND, Command) + STORAGE_PROTOCOL_COMMAND_LENGTH_NVME;
    
data_buffer_offset = p_protocol_cmd->ErrorInfoOffset + p_protocol_cmd->ErrorInfoLength;


p_protocol_cmd->DataToDeviceBufferOffset = data_buffer_offset;
p_protocol_cmd->DataToDeviceTransferLength = data_len;
memcpy(&(p_buffer[data_buffer_offset]), p_user_data, data_len);

p_nvme_cmd->CDW0.OPC = 0x83;
p_nvme_cmd->u.GENERAL.CDW12 = <custom vendor value>;
p_nvme_cmd->u.GENERAL.CDW13 = <custom vendor value>;
p_nvme_cmd->u.GENERAL.CDW14 = <custom vendor value>;
p_nvme_cmd->u.GENERAL.CDW15 = <custom vendor value>;
p_protocol_cmd->CommandSpecific = STORAGE_PROTOCOL_SPECIFIC_NVME_NVM_COMMAND;
/p_nvme_cmd->NSID = 1;

b_retval = DeviceIoControl(
                                device_handle,
                                IOCTL_STORAGE_PROTOCOL_COMMAND,
                                p_buffer,
                                buffer_length,
                                p_buffer,
                                buffer_length,
                                &returned_data_len,
                                NULL
                             );

if (FALSE == b_retval) 
{
  DWORD error = GetLastError();
  NVME_ERROR_INFO_LOG* ptrLog = (NVME_ERROR_INFO_LOG*)(p_buffer + FIELD_OFFSET(STORAGE_PROTOCOL_COMMAND, Command) + STORAGE_PROTOCOL_COMMAND_LENGTH_NVME);
}


In this case, the command always fails with ErrorCode=1117, IO Error.

What we have tried:

1. Setting AVSCC and NVSCC values in the identity Struct returned from the Device (in the device firmware obviously) to 1 and 0 in the Identity command trying to indicate compliance or not to standard NVMe Vendor Commands.

2. Setting NSID (namespace ID in DW0 of the p_nvme_command) to 0, 1 and 0xFFFFFFFF indicating all the possibilities

3. As per NVME spec, setting the p_nvme_cmd->u.GENERAL.CDW10 to 256 indicating a need to transfer 256 DWORDS which is precisely 1024 bytes we are trying to transfer.

None of this worked, we always get an IO_ERRROR error code 1117.

Next, we have discovered that commenting out the following line allows the command to go through

p_protocol_cmd->Flags = STORAGE_PROTOCOL_COMMAND_FLAG_ADAPTER_REQUEST;

however, even though command goes through, no PRP data transfers are done, even though data length and data offset are set in STORAGE_PROTOCOL_COMMAND struct.

We also tried every possible combination mentioned above as well with no luck, no data is ever transferred period.

Device has one namespace, so NSID should be 1 for all non-admin commands.

Everything works fine with the OFA driver, but we want to get this working with standard windows driver since this is important to our customers.

Please let me know what can be the issue, and how to possible resolve it, this is very important for us

clearshot365 on Thu, 27 Jun 2019 15:04:11


Hello,

A dev from the feature team reviewed this and statest that the code looks fine and asked does the vendor specific commands send down has corresponding command effects data from command effects log? If not, storport will block any command to the device as mentioned in https://docs.microsoft.com/en-us/windows/desktop/FileIO/working-with-nvme-devices#pass-through-mechanism

Important:

StoreNVMe.sys and Storeport.sys will block any command to a device if it is not described in the Command Effects Log. 

Hi Cymon,

can you please pass the information i have posted below to your Dev team, its extremely important for us to get this working sooner than later.

thank you