Wow... I've never seen Max 7. I started with Max 8 and then got the ProBoolean add-on.
A few ways to increase the mesh density:
* manually - by selecting the faces where you want to make your boolean, CONTROL + EDGE SELECT (this will select all of the edges from the faces that were selected), then connect the edges together.
* tesselate - select the faces and then press tesselate
* mesh smooth - select the faces (there's a pattern here...
) and then apply a mesh smooth modifier
* turbo smooth - not sure if you've have that or not. Like mesh smooth, only a bit better
Think of the mesh as a monitor. If you have to display a circle and there aren't enough pixels, it will be shaped funny. Not enough resolution to properly define the shape.
Same with boolean and segmentation.