vCloud Director 9.7 – Issue with adding a new role

Introduction

This article will help you solve an issue with vCloud Director 9.7 where it’s not possible to add a new role to the Role Based Access Control section of a tenant.

This morning I wanted to add a new role to my tenant in vCloud Director.
Normally the steps are easy: Navigate to vCloud Director > Administration > Access Control > Roles and choose New.
Here you can select all the responsibilities you want to add to your new role and hit Save.

But not today.. This morning the section where you can select responsibilities to add to your new Role was empty.
In the screenshot below you can see that in my Lab environment everything is OK.

Troubleshooting

I want to share what I did to troubleshoot this issue.

• Open Developer Tools in your browser (F12) and dock it to the right side of the screen

• Navigate to the Roles page under Administration > Access Control and look for errors (displayed in Red)

One of the differences between Production and Lab is the presence of a reverse proxy.
To rule it out I’ve set up a VPN to the environment and configured an entry in my host file (Windows) pointing to the internal IP of the primary vCD cell.
This way I’m not using the HAProxy servers.

To my surprise, I can choose responsibilities for the role now!

HAproxy configuration

Let’s dive into the HAproxy’s..
You can choose to dive into the logs first, or tweak and tune with the bufsize and maxhdr settings immediately.

• SSH to your HAproxy server
• Check the logs for error 502

tail -f /var/log/messages | grep " 502"

• If you want to update the bufsize and maxhdr settings:

# Open de HAproxy config
vi /etc/haproxy/haproxy.cfg
# Settings which worked for me are:
# tune.bufsize 65536
# tune.http.maxhdr 150
# Restart HAproxy services
service haproxy restart

Adding a new role

Now, remove the static entry in your host file and try again via the normal route.
If your change is successful, you can see the responsibilities you want to add to a new Role:

Helpful with solving this issue were this article on stackoverflow and these pieces of documentation below:

Tune.bufsize

Sets the buffer size to this size (in bytes). Lower values allow more sessions to coexist in the same amount of RAM, and higher values allow some applications with very large cookies to work. The default value is 16384 and can be changed at build time.
It is strongly recommended not to change this from the default value, as very low values will break some services such as statistics, and values larger than default size will increase memory usage, possibly causing the system to run out of memory.
At least the global maxconn parameter should be decreased by the same factor as this one is increased.

Inaddition, use of HTTP/2 mandates that this value must be 16384 or more.
If an HTTP request is larger than (tune.bufsize – tune.maxrewrite), haproxy will return HTTP 400 (Bad Request) error. Similarly if an HTTP response is larger
than this size, haproxy will return HTTP 502 (Bad Gateway).

Source: https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#3.2-tune.bufsize 

Tune.http.maxhdr

Sets the maximum number of headers in a request. When a request comes with a number of headers greater than this value (including the first line), it is rejected with a “400 Bad Request” status code.

Similarly, too large responses are blocked with “502 Bad Gateway”.
The default value is 101, which is enough for all usages, considering that the widely deployed Apache server uses the same limit.
It can be useful to push this limit further to temporarily allow a buggy application to work.
The accepted range is 1..32767.

Keep in mind that each new header consumes 32-bits of memory foreach session, so don’t push this limit too high.

Source: https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#3.2-tune.http.maxhdr

PS. Check our other vCloud Director related articles here.