Assume that there are four firms supplying a homogeneous product. The identical cost function given by C(Q)=40Q. If the demand curve for the...

[Minor note: I was a bit confused at first by the notation "U" for the demand curve. Utility and willingness-to-pay really aren't the same thing, and it's dangerous to conflate them in this way. But I assume this is what your book or your instructor used, so I'll use it as well.]

The basic concept of Cournot competition is that each firm optimizes its price and quantity produced while assuming that the other firms' production is fixed; but then, by symmetry, all firms are doing the same, so the resulting equilibrium emerges as the sum of all their choices. Cournot competition is a fairly good approximation for pricing in real-world oligopolies (whereas Bertrand competition is almost completely wrong), but it is still an oversimplification of the complex strategies that oligopoly firms employ.

So, we have our cost function:
C = 40 Q
And we have our demand curve:
U = 100 - Q
I'm more comfortable thinking of this in terms of price P:
P = 100 - Q
This is the marginal willingness-to-pay, the highest price P that customers would be willing to pay and still purchase quantity Q.

Unlike perfect competition, we don't simply set these equal. The firms are trying to set their price above marginal cost, in order to increase profits. If there are few enough firms, they will be able to do so to some extent---but not as much as a monopoly could.

Consider the decision facing firm 1. They will produce q1 and sell at P (the products are not differentiated, so everyone sells at the same price). Other firms will produce some other amounts, q2, q3, q4. All of these must sum to the total quantity produced Q, which is what matters for the overall supply and demand curves.

Q = q1 + q2 + q3 + q4

For reasons that will make sense in a moment, let's call the quantity produced by firms other than firm 1 Q*.

Firm 1 acts as though they can only control q1 and the others are fixed, so they think of Q* as a fixed quantity. Then, they maximize profit:

profit = revenue - cost
0 = marginal revenue - marginal cost

Since firm 1 only controls q1, that is what they take the derivative in terms of:
0 = d/dq1[P*q1] - dC/dq1
0 = P + dP/dq1 * q1 - dC/dq1
Now we can substitute in:
Q = q1 + Q*
P = 100 - (q1 + Q*)
C = 40 (q1 + Q*)

0 = 100 - (q1 + Q*) + (-1) * q1 - 40
2q1 = 60 - Q*

q1 = 30 - Q*/2

[Note: With this linear cost function, the marginal cost for firm 1 doesn't depend on what the other firms do. For other cost functions it might.]

It might seem like we're stuck; we have one equation in two unknowns. But by assuming that all 4 firms reason the same way, we know:

Q = q1 + q2 + q3 + q4
Q = 4q1
Q* = 3q1

Now we can solve:
q1 = 30 - 3q1/2
5q1 = 30
q1 = 6

This is equilibrium industry output:
Q = 4q1 = 24

Price we can get from the demand curve (careful: not the supply curve):
P = 100 - 24 = $76

Profits for each firm:
profit = revenue - cost
profit = P*q1 - C(q1)
profit = (76)(24) - 40(24) = $864