This looked pretty bad, so I took some time to re-examine the test; it generates an elliptic curve using the brainpoolP512t1 curve, which I selected through a rigorous process of running openssl ecparam -list_curves and then just picking one arbitrarily. The certificate is then signed by the server's trusted root authority and then finally passed to the server by the client, at which point the protocol error occurs. There didn't seem to be any particular wrongdoing on my end, so I filed a bug against the service.
The maintainer got back to me pointing to a bug filed against OpenSSL in which the project's maintainer responded that it was not possible to use the brainpoolP512t1 curve as it had not been assigned a number by the Internet Assigned Numbers Authority (IANA). But, wait a second, it had worked on the older, 1.0.1 series, and upon closer inspection, the filer was also using the 1.0.1 series, and their error messages were different than mine. After digging a little deeper I realized that they were using EC for the server certificate, while I was using it for the client. From the ticket I was able to reproduce their version of the issue on both the 1.0.1 series and the 1.1.0 series with the following commands:
openssl ecparam -out eckey.pem -name brainpoolP512t1 -genkey openssl req -x509 -new -key eckey.pem -out eccert.pem -subj "/CN=EC-SHA512/" openssl s_server -accept 415 -cert eccert.pem -key eckey.pem -4 openssl s_client -connect 127.0.0.1:415
This did not, however, explain why I had ever been able to connect using EC in the client certificate and my brain was starting to turn to mush trying to figure out what facts might cause such inconsistent behaviour. Configuring the service was proving to be tricky and I wanted to test against a known-correct implementation, so I decided to see if I could reproduce the client error by the command-line OpenSSL utilities. I began by generating keys and self-signed certificates:
openssl ecparam -out eckey.pem -name brainpoolP512t1 -genkey openssl req -x509 -new -key eckey.pem -out eccert.pem -subj "/CN=EC-SHA512/" openssl genrsa -out rsakey.pem 4096 openssl req -key rsakey.pem -new -x509 -days 7200 -sha512 -out rsacert.pem -subj '/CN=RSA/'
...then attempted to connect to the server while using EC in the client certificate:
openssl s_server -accept 415 -cert rsacert.pem -key rsakey.pem -4
openssl s_client -cert eccert.pem -key eckey.pem -connect 127.0.0.1:415
To my great consternation, this worked on both OpenSSL series. After some colourful language and gnashing of teeth I took a look at the man page of s_server and found a -verify and -verify_return_error option, and, after making sure I had set the server's Certificate Authority properly via the -CAfile eccert.pem option, tried again and, finally, was able to reproduce the behaviour that occurred on the service: The EC client certificate would break on the OpenSSL 1.1.0 series with a protocol error but work in the 1.0.1 series.
Since this behaviour was unexpected, I decided to ask the OpenSSL maintainers about it by filing an issue about it on their GitHub page. I was informed that "...implicit support for the 't' versions was removed ..." and that "...If the curve does not have an IANA number, it cannot work in TLS". Well, this doesn't quite explain to me how the curve ever worked in the old series, but I didn't feel up to analyzing the protocol and source code in order to find out. They also suggesting using the -named_curve and -curves flag, but neither option worked for me.
As brainpoolP512p1 was now clearly not portable, the next step was to try a different curve. I chose the brainpoolP512r1 curve mentioned in the first OpenSSL bug, and this worked for both series of the library. Thus my noggin-melting problem was solved by changing a single character. Perhaps in the future I'll be more leery of making arbitrary selections, but in this case a quick Internet search beforehand didn't reveal anything digestible, so probably not.