Mouse interaction and custom cursor

Recently I've been working quite a bit with replacing the mouse cursor with custom graphics. And usually there are a lot of custom behaviour to it, similar to, play some animation when LMB is pressed, the custom cursor should be disabled while over a specific area (buttons). And last but not least, something specific should happen when user presses LMB inside a specific area for example attaching new movie clips while LMB is down etc.

In the above scenario, this is the best solution I've come up with this far. Imagine you have a "container" that encapsulates every movie clip that you will be using, you then listen to mouse-press and starts listening to enter-frame messages:

function init:
_container.addEventListener(MouseEvent.MOUSE_DOWN, onMouseDown);
_container.addEventListener(Event.ENTER_FRAME, onEnterFrame);
_cursor.visible = true;
_cursor.startDrag(true);
Mouse.hide();

then inside the function onMouseDown (observe the stage-reference):

_isMouseDown = true;

_container.removeEventListener(MouseEvent.MOUSE_DOWN, onMouseDown);

_container.stage.addEventListener(MouseEvent.MOUSE_UP, onMouseUp);

and inside the function onMouseUp

_isMouseDown = false;

_container.addEventListener(MouseEvent.MOUSE_DOWN, onMouseDown);

_container.stage.removeEventListener(MouseEvent.MOUSE_UP, onMouseUp);

and finally the onEnterFrame-function, using a hitbox (subpart of "container" where the mouse-cursor should be replaced with custom-cursor) (if you are curious about the hitTestMouse-function please see my previous post):

{

if (hitTestMouse(_hitbox)) { 

    _cursor.visible = true;

    Mouse.hide();

} else { 

    _cursor.visible = false;

    Mouse.show();

    return; //outside of bounds

} 

if(_isMouseDown == false) { 

     return; 

}

/* depending on your solution, you could either create an awesome shape for hit test (_hitBox), or you could use a simple rectangle and then do more escape routes here in code with something similar to if(hitTestMouse(_button2)) useCustomCursor(false) and then return;

*/

//create and attach your custom movieclip here

var mc:MovieClip = new _eraserDef();

mc.x = _container.mouseX;

mc.y = _container.mouseY;

_container.foo.addChild(mc);

}

This will make it possible to replace mouse cursor with custom graphics while over a specific area (hitbox) and then restore regular cursor while hovering above buttons or outside hitbox and "place symbols" or "draw" while inside the hitbox. 

Don't remember to kill all event-listeners, stopdrag on _cursor, and to restore mouse when you're done with what you're gonna do.

Proper hitTest in AS3

After many ugly attempts at handling hit tests in AS3 I finally managed to solve what I was looking for. The thing I forgot to consider was to use the global mouse-coordinates.

public function hitTestMouse(hitarea:DisplayObject):Boolean {
    return hitarea.hitTestPoint(hitarea.stage.mouseX, hitarea.stage.mouseY, true);
}

The last parameter (shapeFlag) would decide if the hit test are checked any actual shapes inside the hitarea that the mouse is hovering above.

Take the picture below as an example, say that the two buttons "autoskrap" and "vinstkontroll" are inside a movie clip (marked as the grayish box) if the shapeFlag is set to false (default) the hitTestMouse-function would return true whenever the mouse were somewhere inside the gray box. If the shapeFlag is set to true it would only return true if the mouse were over the actual buttons.