I just started using IDA and OllyDbg, although I am experienced developer but before this I was reverse engineering only Java and .Net code which is much more easy task comparing of what I need to do now.I have an application which has some buttons that are not accessible by Spy++ and WinInspector and all other tools that I tried. Spy++ sees only the selectorclass which is the whole panel where buttons are placed. I've managed to find where the WndProc of this selector is and even the method wich takes hwnd of SelectorControl from GetPropA method, but I have trouble identifying the click itself, so i don't know where the actual button is clicked and where there the app knows what exact button is clicked and what handler to use. I would like to ask for some tips which are maybe useful to know but are not known by newbies like me. What are strategies that can be used here?Maybe there are some ways to identify what framework was used, so I can play with that framework to know what to look for.
Custom UI buttons
ida;ollydbg;c++
If the surrounding window seems to be a genuine window, but the individual buttons are not, I'd assume thatThe surrounding window displays a bitmap, or does whatever else is needed to display the individual buttonsThe surrounding window reacts on mouse clicksAnd, the surrounding window determines itself which button is used by looking at the mouse coordinates at the time of the mouse click.So, what we want to do is, start the program in a debugger, and use a breakpoint that is triggered when the mouse click occurs. The problem is, the window procedure gets called a lot of times, with all kind of messages, while you want to break on some of them (for example, WM_LBUTTONDOWN), but not on most of the others (for example, WM_PAINT).Ollydbg can do this directly - place a conditional breakpoint that triggers depending on the window message.If, for whatever reason, you can't use that feature, you need to disassemble (and partially understand) the window procedure. It will receive 4 parameters on the stack, the 2nd of which is the window message (see Using Window Procedures). The window procedure will, at some point, compare this parameter to the values of well-known window message IDs. You might be interested in WM_LBUTTONDOWN (0x0201) or one of the other Mouse Input Events). Check where the code branches to when it receives one of those interesting messages, and place a breakpoint on that branch.After placing a suitable breakpoint, run your application in the debugger, click the mouse, and hope the debugger actually hits the breakpoint. If it doesn't, check which mistake you've made.Single-Step your code. It will probably fetch parameter 4 from the stack, which encodes the X and Y coordinates of the mouse, and compare these parameters to something - chances are, there's an array of structures that defines rectangles and jump addresses, and a loop that checks against each entry.Once you've found that, you're finished.Sounds too easy? It probably is, because, depending on your framework, the actual comparisons might be buried deeply in stuff that was a bunch of C++ classes in the original source, and there might be various calls, some of them indirect (in the case of C++ class methods) until you get to the interesting position.Sometimes it's easier to use Ollydbg to resolve this. Single-Step the source after your breakpoint, using step over to step over function calls. Whenever you see something interested happened within the function call, note the address. Restart your program, run to your breakpoint, run to the address you noted, and step into the function this time. This will allow you to put the finger on the interesting code relatively quickly, while stepping over all the strcmp() and malloc() and WriteALotOfStuffToSomeFileIfDebuggingIsOn() functions.Sometimes it's easier to use IDA. Generate a call tree from your window procedure, focus on the first 3-5 levels. Search your code for occurrences of magic numbers, like the 0x0201 above. Check if one of these is a CMP XX, 0x0201 in one of the functions in your call tree. This might be just the position you're looking for.Make a screen shot, and paste it into some image viewer program (I like Irfanview for that). Get the pixel coordinates of the button within the frame window. Search, in the data section, for those values, surrounded by what looks like a table of similar data. This might be your coordinate/jump table. If you find something, place a hardware breakpoint on it to find out when that memory gets accessed, or use the IDA cross reference list to get respective addresses in your code.