SSL_Handshake()--Initiate the SSL Handshake Protocol

Syntax


 #include <qsossl.h>


 int SSL_Handshake(SSLHandle* handle,
                   int how)

  Service Program Name: QSOSSLSR

  Default Public Authority: *USE

  Threadsafe: Yes

The SSL_Handshake() function is used by a program to initiate the SSL handshake protocol. Both the client and the server program must call the SSL_Handshake verb in order to initiate the handshake processing.

Parameters

SSLHandle* handle (input/output)

The pointer to an SSLHandle for an SSL session. An SSLHandle is a typedef for a buffer of type struct SSLHandleStr. In <qsossl.h>, struct SSLHandleStr is defined as the following:

struct SSLHandleStr {                         /* SSLHandleStr               */
   int              fd;                       /* Socket descriptor          */
   int              createFlags;              /* SSL_Create flags value     */
   unsigned         protocol;                 /* SSL protocol version       */
   unsigned         timeout;                  /* Timeout value in seconds   */
   unsigned         char cipherKind[3];       /* Current 2.0 cipher suite   */
   unsigned         short int cipherSuite;    /* Current 3.0 cipher suite   */
   unsigned         short int* cipherSuiteList; /* List of cipher suites    */
   unsigned int     cipherSuiteListLen;       /* Number of entries in
                                                 the cipher suites list     */
   unsigned         char* peerCert;           /* Peer certificate           */
   unsigned         peerCertLen;              /* Peer certificate length    */
   int              peerCertValidateRc;       /* Return code from
                                                 validation of certficate   */
   int             (*exitPgm)(struct SSLHandleStr* sslh);
                                              /* Authentication exit
                                                 program called when a
                                                 certificate is received
                                                 during SSL handshake       */
};

The fields within the SSLHandle structure as pointed to by handle are defined as follows:

int fd (input)

The socket descriptor of the connection for which the SSL handshake protocol is to be performed. This field was initialized by a prior SSL_Create() API.

int createFlags (input)

Whether or not the SSL protocol is to be used. If the field specifies a value that does not include the SSL_ENCRYPT flag, then this function will return success without performing the SSL handshake protocol. This field was initialized by a prior SSL_Create() API.

unsigned int protocol (input/output)

The type of SSL handshake protocol to be performed. The protocol(s) that are acceptable as the handshake protocol for this job. The following values may be specified for protocol and are defined in <qsossl.h>.

SSL_VERSION_CURRENT     0 (TLS with SSL Version 3.0 and SSL
                           Version 2.0 compatibility)
SSL_VERSION_2           2 (SSL Version 2.0 only)
SSL_VERSION_3           3 (SSL Version 3.0 only)
TLS_VERSION_1           4 (TLS Version 1 only)
TLSV1_SSLV3             5 (TLS Version 1 with SSL
                           Version 3.0 compatibility)

Upon return, this field will be set to reflect the protocol version actually negotiated. If the createFlags field specifies a value that does not include the SSL_ENCRYPT flag, then this field will be unchanged from its input value.

unsigned timeout (input)

The approximate number of seconds to wait for the SSL handshake protocol to complete. A value of 0 indicates to wait forever for the handshake to complete.

unsigned char cipherKind[3] (output)

The cipher kind (which is the SSL Version 2.0 cipher suite) negotiated by the handshake.

unsigned short int cipherSuite (output)

The cipher suite type negotiated by the handshake.

unsigned short int* cipherSuiteList (input)

A pointer to a cipher specification list that is to be used during the handshake negotiation for this SSL session. This list is a string of concatenated cipher specification values. Each cipher specification is an unsigned short integer value. Any value provided will override, for this SSL session, the default cipher specification list provided by a previous SSL_Init() API or SSL_Init_Application() API . The valid cipher suites allowed are defined in <qsossl.h>. A value of NULL indicates one of the following:

Use the cipher specification list provided by a previous SSL_Init() API or SSL_Init_Application() API
Use the system default cipher specification list if the previous SSL_Init() API or SSL_Init_Application() API did not provide a cipher specification list

unsigned int cipherSuiteListLen (input)

The number of cipher suite entries specified in the list pointed to by the cipherSuiteList field.

unsigned char* peerCert (output)

The pointer to the certificate received from the peer system. For a client, this is a pointer to the server's certificate. For a server with client authentication enabled, this is a pointer to the client's certificate. For a server without client authentication this pointer value remains unchanged.

unsigned peerCertLen (output)

The length of the certificate pointed to by the peerCert field.

int (*exitPgm)(SSLHandle* sslh) (input)

A pointer to a user supplied function that is called whenever a certificate is received during handshake processing. The exitPgm will be passed the pointer to the handle, which could include the peer's certificate. The exitPgm returns a nonzero value if the peer's certificate is accepted. The return of a zero value by the exitPgm will cause the handshake processing to fail. Users of this function do not need to provide an exit program. The pointer should be a NULL value if there is not a user-supplied exit program. The exit program should be written in a threadsafe manner.

int how (input)

The type of SSL handshake to be performed. The following values may be specified for handshake type and are defined in <qsossl.h>.

SSL_HANDSHAKE_AS_CLIENT (0)	Perform the handshake as a client.
SSL_HANDSHAKE_AS_SERVER (1)	Perform the handshake as a server.
SSL_HANDSHAKE_AS_SERVER_WITH_CLIENT_AUTH (2)	Perform the handshake as a server with client authentication.
SSL_HANDSHAKE_AS_SERVER_WITH_OPTIONAL_CLIENT_AUTH (3)	Perform the handshake as a server with optional client authentication.

Authorities

Authorization of *R (allow access to the object) to the key ring file and its associated files is required.

Return Value

The SSL_Handshake() API returns an integer. Possible values are:

[0]: Successful return
[SSL_ERROR_BAD_CERTIFICATE]: The certificate is bad.
[SSL_ERROR_BAD_CERT_SIG]: The certificate's signature is not valid.
[SSL_ERROR_BAD_CERTIFICATE]: The certificate is bad.
[SSL_ERROR_BAD_CIPHER_SUITE]: A cipher suite that is not valid was specified.
[SSL_ERROR_BAD_MAC]: A bad message authentication code was received.
[SSL_ERROR_BAD_MALLOC]: Unable to allocate storage required for SSL processing.
[SSL_ERROR_BAD_MESSAGE]: SSL received a badly formatted message.
[SSL_ERROR_BAD_PEER]: The peer system is not recognized.
[SSL_ERROR_BAD_STATE]: SSL detected a bad state in the SSL session.
[SSL_ERROR_CERTIFICATE_REJECTED ]: The certificate is not valid or was rejected by the exit program.
[SSL_ERROR_CERT_EXPIRED]: The validity time period of the certificate is expired.
[SSL_ERROR_CLOSED]: The SSL session ended.
[SSL_ERROR_IO]: An error occurred in SSL processing; check the errno value.
[SSL_ERROR_NO_CERTIFICATE]: No certificate is available for SSL processing.
[SSL_ERROR_NO_CIPHERS]: No ciphers available or specified.
[SSL_ERROR_NO_INIT]: SSL_Init() was not previously called for this job.
[SSL_ERROR_NOT_TRUSTED_ROOT]: The certificate is not signed by a trusted certificate authority.
[SSL_ERROR_PERMISSION_DENIED]: Permission was denied to access object.
[SSL_ERROR_SSL_NOT_AVAILABLE]: SSL is not available for use.
[SSL_ERROR_UNKNOWN]: An unknown or unexpected error occurred during SSL processing.
[SSL_ERROR_UNSUPPORTED_CERTIFICATE_TYPE]: i5/OS does not support the certificate's type.
[SSL_ERROR_UNSUPPORTED_CERTIFICATE_TYPE]: i5/OS does not support the certificate's type.

Error Conditions

When the SSL_Handshake() API fails with a return code of [SSL_ERROR_IO], errno can be set to:

[EACCES]

Permission denied.

[EBADF]

Descriptor not valid.

[EBUSY]

Resource busy.

[ECONNRESET]

A connection with a remote socket was reset by that socket.

[EDEADLK]

Resource deadlock avoided.

[EFAULT]

Bad address.

The system detected an address that was not valid while attempting to access the handle parameter or one of the address fields in the handle parameter.

[EINTR]

Interrupted function call.

[EINVAL]

Parameter not valid.

This error code indicates one of the following:

The socket_descriptor type is not SOCK_STREAM or address family is not AF_INET or AF_INET6.
One of the parameters passed is not valid or is NULL.
The protocol field contains a value that is not valid.

[EALREADY]

Operation already in progress.

An SSL_Handshake() API has already been previously successfully completed.

[EIO]

Input/output error.

[ENOBUFS]

There is not enough buffer space for the requested operation.

[ENOTCONN]

Requested operation requires a connection.

This error code indicates one of the following:

The socket_descriptor is not for a socket that is in a connected state.
The socket_descriptor has not had SSL support enabled.

[ENOTSOCK]

The specified descriptor does not reference a socket.

[EPIPE]

Broken pipe.

[ETIMEDOUT]

A remote host did not respond within the timeout period.

[EUNATCH]

The protocol required to support the specified address family is not available at this time.

[EUNKNOWN]

Unknown system state.

Any errno that can be returned by send() or recv() can be returned by this API. See Sockets APIs for a description of the errno values they return.

If an errno is returned that is not in this list, see Errno Values for UNIX-Type Functions for a description of the errno.

Error Messages

Message ID	Error Message Text
CPE3418 E	Possible APAR condition or hardware failure.
CPF9872 E	Program or service program &1 in library &2 ended. Reason code &3.
CPFA081 E	Unable to set return value or error code.

Usage Notes

The SSL_Handshake() function is only valid on sockets that have an address family of AF_INET or AF_INET6 and a socket type of SOCK_STREAM. If the descriptor pointed to by the handle structure parameter does not have the correct address family and socket type, [SSL_ERROR_IO] is returned and the errno value is set to EINVAL.
The SSL_Handshake() function can be called only one time per SSL session. If a secondary call of SSL_Handshake() occurs within the same established SSL session, then it will fail and the errno will be set to [einval].
A successful SSL_Init() or or SSL_Init_Application() API and a successful SSL_Create() API must be called prior to an SSL_Handshake() API. The SSL_Init() API or SSL_Init_Application() API is used to establish a certificate and private key for either of the following:
- A successful handshake as a server
- A successful handshake as a client when connected to a server performing client authentication
The SSL_Create() API is used to enable SSL support for the specified socket descriptor.
When doing SSL_Handshake() with a how parameter value of SSL_HANDSHAKE_AS_SERVER, SSL_HANDSHAKE_AS_SERVER_WITH_CLIENT_AUTH, or SSL_HANDSHAKE_AS_SERVER_WITH_OPTIONAL_CLIENT_AUTH, the cipherSuite value (if TLS_VERSION_1 or SSL_VERSION_3 protocol) or the cipherKind (if SSL_VERSION_2 protocol) will be the first cipher found in the ordered cipherSuiteList list that was also found in the cipher list provided by the client during the SSL handshake.
When doing SSL_Handshake() with a how parameter value of SSL_HANDSHAKE_AS_CLIENT, the cipher specification list will be sent to the server in the client hello in the order found in the cipherSuiteList, however the value from that list that is negotiated for the cipherSuite value (if TLS_VERSION_1 or SSL_VERSION_3 protocol) or the cipherKind (if SSL_VERSION_2 protocol) is determined by the server policy.

Related Information

SSL_Create()--Enable SSL Support for the Specified Socket Descriptor
SSL_Destroy()--End SSL Support for the Specified SSL Session
SSL_Init()--Initialize the Current Job for SSL
SSL_Read()--Receive Data from an SSL-Enabled Socket Descriptor
SSL_Write()--Write Data to an SSL-Enabled Socket Descriptor

API introduced: V4R3

Top | UNIX-Type APIs | APIs by category