Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Troubleshooting the API execution error #112

Open
shsarava opened this issue Jun 21, 2020 · 13 comments
Open

Troubleshooting the API execution error #112

shsarava opened this issue Jun 21, 2020 · 13 comments

Comments

@shsarava
Copy link

@shsarava shsarava commented Jun 21, 2020

Trying to add a server to the backend, with the following steps. Not sure why the transaction fails.

Got a new Transaction

curl -X POST --user admin:admin -H "Content-Type: application/json" http://localhost:5555/v2/services/haproxy/transactions?version=1
{"_version":1,"id":"57768891-56ea-436b-b85e-f7450dd616c8","status":"in_progress"}

Adding a new backend server to the existing backend

curl -X POST --user admin:admin -H "Content-Type: application/json" -d '{"name": "server2", "address": "10.0.0.40", "port": 80, "check": "enabled", "maxconn": 30, "weight": 80}' "http://localhost:5555/v2/services/haproxy/configuration/servers?backend=app&transaction_id=57768891-56ea-436b-b85e-f7450dd616c8"
{"address":"10.0.0.40","check":"enabled","maxconn":30,"name":"server2","port":80,"proxy-v2-options":null,"weight":80}

Committing the transaction

curl -X PUT --user admin:admin -H "Content-Type: application/json" "http://localhost:5555/v2/services/haproxy/transactions/57768891-56ea-436b-b85e-f7450dd616c8"

Getting an Error

{"code":400,"message":"14:** ERR transactionId=57768891-56ea-436b-b85e-f7450dd616c8 "}

Am I missing any needed steps ? How can I troubleshoot the error

@mjuraga
Copy link
Collaborator

@mjuraga mjuraga commented Jun 22, 2020

This means that checking the the transaction file syntax has failed. Check what you have set up for your -b --haproxy-binary (default: haproxy) set up. Then you can check the failed transactions directory set up in your dataplaneapi (default: /tmp/haproxy/failed), and look for your failed transaction config file (suffix 57768891-56ea-436b-b85e-f7450dd616c8).

See what's wrong with the config and try running <haproxy-binary set up> -c -f <failed transaction file> to see why haproxy checks fails with that file.

Also try running it as the same user that is running the dataplaneapi.

@shsarava
Copy link
Author

@shsarava shsarava commented Jun 23, 2020

Thank you for the quick response. I tried to use the same failed config with haproxy binary and it ran successfully. I tried to run the dataplaneapi with trace logging but didn't get much output when the error happened. Not sure what could be the issue..

@mjuraga
Copy link
Collaborator

@mjuraga mjuraga commented Jun 23, 2020

Can you paste here how you run your dataplaneapi? What CLI options do you use there?

@shsarava
Copy link
Author

@shsarava shsarava commented Jun 23, 2020

Processes I see running

root 7702 1 0 Jun21 ? 00:00:00 sudo ./dataplaneapi --port 5555 -b /usr/sbin/haproxy -c /etc/haproxy/haproxy.cfg -d 5 -r systemctl restart haproxy -s systemctl restart haproxy -u datapla

root 7703 7702 0 Jun21 ? 00:00:20 ./dataplaneapi --port 5555 -b /usr/sbin/haproxy -c /etc/haproxy/haproxy.cfg -d 5 -r systemctl restart haproxy -s systemctl restart haproxy -u dataplaneapi

CLI I used to run the DataplaneAPI

sudo ./dataplaneapi --port 5555 -b /usr/sbin/haproxy -c /etc/haproxy/haproxy.cfg -d 5 -r "systemctl restart haproxy" -s "systemctl restart haproxy" -u dataplaneapi -t /tmp/haproxy --userlist api &

@mjuraga
Copy link
Collaborator

@mjuraga mjuraga commented Jun 23, 2020

And when you try to execute:

sudo /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c

What is the output?

@shsarava
Copy link
Author

@shsarava shsarava commented Jun 23, 2020

I am actually using systemd to run haproxy and the service config file is below

ec2-user@ip-10-0-0-214 ~]$ cat /etc/systemd/system/haproxy.service
[Unit]
Description=HAProxy Load Balancer
After=network.target
[Service]
Environment=LD_LIBRARY_PATH=/usr/local/openssl-1.1.1c/lib/
Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/run/haproxy.pid" "EXTRAOPTS=-S /run/haproxy-master.sock"
ExecStartPre=/usr/local/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS
ExecStart=/usr/local/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $EXTRAOPTS
ExecReload=/usr/local/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS
ExecReload=/bin/kill -USR2 $MAINPID
KillMode=mixed
Restart=always
SuccessExitStatus=143
Type=notify
[Install]
WantedBy=multi-user.target

@mjuraga
Copy link
Collaborator

@mjuraga mjuraga commented Jun 23, 2020

It seems your haproxy is located at /usr/local/sbin/haproxy

Try running dataplaneapi with -b /usr/local/sbin/haproxy

@shsarava
Copy link
Author

@shsarava shsarava commented Jun 23, 2020

Made the change

root 10495 10275 0 14:57 pts/2 00:00:00 sudo ./dataplaneapi --port 5555 -b /usr/local/sbin/haproxy -c /etc/haproxy/haproxy.cfg -d 5 -r systemctl restart haproxy -s systemctl restart haproxy -u dataplaneapi -t /tmp/haproxy --userlist api

root 10496 10495 0 14:57 pts/2 00:00:01 ./dataplaneapi --port 5555 -b /usr/local/sbin/haproxy -c /etc/haproxy/haproxy.cfg -d 5 -r systemctl restart haproxy -s systemctl restart haproxy -u dataplaneapi -t /tmp/haproxy --userlist api

Same steps of adding a server to backend and getting the same error

[ec2-user@ip-10-0-0-214 ~]$ curl -X POST --user admin:admin -H "Content-Type: application/json" http://localhost:5555/v2/services/haproxy/transactions?version=1
{"_version":1,"id":"f3a8b275-c435-428d-a17e-b7441d8effbb","status":"in_progress"}

[ec2-user@ip-10-0-0-214 ~]$ curl -X POST --user admin:admin -H "Content-Type: application/json" -d '{"name": "server3", "address": "10.0.0.41", "port": 80, "check": "enabled", "maxconn": 30, "weight": 80}' "http://localhost:5555/v2/services/haproxy/configuration/servers?backend=app&transaction_id=f3a8b275-c435-428d-a17e-b7441d8effbb"
{"address":"10.0.0.41","check":"enabled","maxconn":30,"name":"server3","port":80,"proxy-v2-options":null,"weight":80}

[ec2-user@ip-10-0-0-214 ~]$ curl -X PUT --user admin:admin -H "Content-Type: application/json" "http://localhost:5555/v2/services/haproxy/transactions/f3a8b275-c435-428d-a17e-b7441d8effbb"
{"code":400,"message":"14: ERR transactionId=f3a8b275-c435-428d-a17e-b7441d8effbb "}

@shsarava
Copy link
Author

@shsarava shsarava commented Jun 25, 2020

I used 2.0.0 and it worked. 2.0.3 seems to have a bug .. even the logging doesn't work..

HAProxy Data Plane API v2.0.0 448d36e

Build from: git@github.com:haproxytech/dataplaneapi.git
Build date: 2020-04-27T15:28:42Z

@andyallan
Copy link

@andyallan andyallan commented Jul 6, 2020

We had the same issue. With 2.0.3 every transaction commit now fails.

Rolling back to 2.0.2 solves it for us.

@mjuraga
Copy link
Collaborator

@mjuraga mjuraga commented Jul 10, 2020

related to #119

@mjuraga
Copy link
Collaborator

@mjuraga mjuraga commented Jul 16, 2020

What is your haproxy version? It seems that older versions of haproxy report results of -c differently. Please check #119 for more details.

@andyallan
Copy link

@andyallan andyallan commented Jul 24, 2020

Sorry for the delay.
HA-Proxy version 2.0.15-1ppa1~bionic 2020/06/13 - https://haproxy.org/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.