Alright then, let me word it this way: If you went back in time a few million/billion years and squashed some prehistoric bugs, would our future currently be different? I doesn't mean all of a sudden humans wouldn't exist or anything quite that dramatic, but history would have changed, and thus the future would have had to.
Now, take a slightly less extreme example: If you went back in time to around the time you were convcieved, and even if you didn't actually talk to anyone or interact with anything sentient, would the future have changed? Certainly, though not very dramatically. However, none the less, the future would have changed in some minute way, and your future self would be getting in his time machine in a -different- future than the one you originally came from.
The second you propose Person A going back in time, it is the second that the timeline from the eyes of Person A and the timeline of his current present become seperate. What has happened in the current timeline from the eyes of Person A is completely moot, as he has entered a seperate timeline as soon as he is back in time, whereas the "current timeline" from which he originated is simply and suddenly missing a Person A, because that timeline cannot exist if a Person A also existed in its past. And before you state, "How can a few footprints left on the ground mean that Person A isn't born", keep in mind that even though another person exactly like Person A most likely will be born, it still isn't the exact same future, and thus he -isn't- Person A; He's simply identical.