$f(x) = x − n$
and let $g_m (x) = f(f(\ldots f (\lfloor x \rfloor) \ldots))$
where $f$
is applied $m$
times. Find $\int_0^{mn} g_m(x)\ dx$
as a function of $n$
and $m,$
where $n$
and $m$
are positive integers.

 Sketch the graph of
$y=\lceil x \rceil ^ 2.$
 Find an expression for the sum
$1^2+2^2+\ldots+n^2.$
 Evaluate
$\int_{2}^{2} (x1)\ dx.$
 Sketch the graph of

Have you tried sketching
$g_m(x)$
for simple values of$n$
and$m?$

Why not start with
$f(\lfloor x \rfloor)$
and then successively add more applications of$f?$

Could you quantify any emerging patterns?

… specifically, the number and height of the increasing and decreasing staircases pattern.

Consider the bounds of the integral,
$0$
to$mn.$
How does this relate to the widths of the staircase pattern? 
You may have different expressions for the two cases, based on the parity of
$m.$

Have you tried finding the area of each individual increasing and decreasing staircase section?

A sketch of the graph is extremely useful for this question; a sample plot is given below.
First we note that
$\lfloor x\rfloor$
is an increasing staircase that has the first flat piece at$y=0.$
Then applying$f$
once subtracts$n$
and flips the negative part up, which gives us a decreasing staircase from$y=n$
down to$y=1,$
then an increasing staircase from$y=0$
to infinity. A second application of$f$
again subtracts and flips, but this time we get an increasing staircase from$0$
to$(n1),$
then a decreasing staircase from$n$
to$1,$
then an increasing staircase to infinity. A total of$m$
applications of$f$
will form$\frac{m}{2}$
bumps, each of which is$2n$
units wide. Hence the integral upper bound$mn$
tells us to find the area under the bumps only.Let’s label the value of the integral
$I$
and for simplicity, split up the problem into two cases: where$m$
is odd and where it is even. For$m=2k,$
we have$k$
increasing step patterns from$0$
to$(n1)$
and$k$
decreasing step patterns from$n$
to$1.$
Since the steps are of unit width, we sum these arithmetic progressions:$I=k\frac{n(n1)}{2}+k\frac{n(n+1)}{2}=kn^2=\frac{mn^2}{2}.$
For
$m=2k+1,$
we have$(k+1)$
decreasing step patterns from$n$
to$1$
and$k$
increasing step patterns from$0$
to$(n1).$
Hence our expression for the total area becomes$I=(k+1)\frac{n(n+1)}{2}+k\frac{n(n1)}{2}=kn^2+\frac{n(n+1)}{2}=\frac{mn^2+n}{2}.$
Plot of
$g_m(x)$
for$n=3$
and$m=4:$