It all depends on the level of support from your spouse. If she is willing to tighten the belt along with you, it may be the best thing you ever did.
How long have you been doing the side business? Do you have a history of sales/net profit to which you can refer as a baseline for your budget? .
I took a similar leap back in 2000 when I left a good paying, steady job that I had been at for about 12 years or so to pursue what had been my sideline. I had been doing it (brokering printing) on the side for about 3 years and had a pretty good income stream going. I took the leap and then by 2003 or so, I made more money than I ever thought I would. Printing started to get lean about that time, so I transitioned into owning a sign shop and continued brokering printing.
I sold that business and took another "real job" in 2006. It was a decent paying job, but being the whole idea of my own boss kept calling. I left that job and its $75K salary (with the 100% support of my wife) and moved back to Florida to start over again with a buddy of mine in a different industry. After a few months, I could see his motivation was not the same as mine (his wife made really good money and did not have any kids), so I started another sign company. Nowadays, I hardly do any print brokering and 95% of my business is from signs. Being hungry is a great motivator. So is having a wife and 3 kids to feed. Thankfully, my wife and I are still together, although it was not easy when I left my job, moved back to Florida and had to hustle and work 6-7 days a week doing signs plus restoring cars just to make ends meet.
I'd say, if your wife supports you and you have a passion for the work, Go For It! I never have a single day where I "just don't feel like going to work" like I did when I worked a "regular job."