From an Agent's point of view, only a reply or a public note will be classified as a response. These are the ones that could be viewed on a ticket by a customer and the response time would be calculated based on this.
SLA policy and timer is tied to the response sent by an agent on a ticket. A private note isn't visible to the customer and hence wouldn't be considered to be a response/first response of that of an agent.