"How do I do this?"
- Who defined the network for each computer: you, or someone else?
- Start > Control Panel > Network Setup Wizard
There is much tutorial information in your Windows "Help and Support". Have you looked at that? If you don't have the fundamentals in mind then a set of step-by-step instructions is a waste of time.
"I have absolutely no experience with router talk."
- Fine; you don't want to change the router anyway. There is too much temptation to twitch things that could mess up the whole bailiwick.
- But DHCP is not just 'router talk' - it is basic to IP connectivity. Generally it is the default in your network definitions. Don't worry about it.
Please do some homework; then ask questions for clarification.
I will note that in the Setup Wizard that the wording can be misinterpreted. The area is 'network connected through a computer'. If you select this (you don't want to!) the network connections are created for ICS - Internet Connection Sharing. This means that the computer is doing the router function; it requires two network adapters on the computer. This may be what was (mis-)defined. Follow the instructions on the referenced post to get it right.
UserAgent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5