Dwarf (pygmies) octos can be entertaining and delightful pets IF you happen to get one that is active and IF you are often up after midnight. I know one of our members had a dwarf in Canada and her experience is journaled here. There are two other journals (all three members purchased egg baring females), gholland's and my own, that you might wish to read if the dwarfs intrest you but they are not the typically kept pet.
If it was a true pygmy octopus a 24 gallon should have been plenty for it. You may have thought it was unhappy because it was inactive or sluggish, but that's the way a pygmy will be unless the lights are out.
Back on topic though a 40 should be okay for 3 Sepia Bandensis if you have premium filteration and at least 1 female. I would not exceed 3 in that size tank.
I kept three S. bandensis in a 29 gallon for the first few months of their life. Then I moved them to a 55 gallon. I think they were 5 months old when I moved them and they lived to be 14 months old in the 55 gallon. In my opinion, 55 gallons is a better size for S. bandensis. While they can certainly survive in a smaller tank, they seemed to thrive in my larger tank. I don't know if S. bandensis are territorial, but my three had favorite spots that they liked to "hang out" in the tank, that were separate from each other. If you happen to get two males, you are definitely going to want some more room for them.