Calculate Budget Hours versus Actual Hours only when budget hours has a value

I want to calculate number of hours left on a project where say we budgeted for 100 hours and actual hours (from time sheets) could be 80 so we would have a results of 20 which is easy to do however I want to add a condition to only calculate the two fields if there is actually a budget number hours available?

I was trying something like:

SUMIF([Budgeted Hours] != "", [Budgeted Hours] - [Actual Hours])

Maybe I am using the wrong function for this. I tried COUNTIF as well.

Any suggestion on how to get this done would be appreciated.

Thanks Paul