This paper reviews toxicological data from human and animal studies regarding neem derived pesticides. I don't have access to the full paper but in the abstract it says that
The non-aqueous extracts appear to be the most toxic neem-based products, with an estimated safe dose (ESD) of 0.002 and 12.5 microg/kg bw/day. Less toxic are the unprocessed materials seed oil and the aqueous extracts (ESD 0.26 and 0.3 mg/kg bw/day, 2 microl/kg bw/day respectively). Most of the pure compounds show a relatively low toxicity (ESD azadirachtin 15 mg/kg bw/day).
and
This leads to the conclusion that, if applied with care, use of neem derived pesticides as an insecticide should not be discouraged.
However, in very large doses neem oil can be toxic.
According to this recent paper the LD50 value of neem oil is 31.95g/kg but the abstract doesn't say if this is for humans or a specific type of animal.
However, since this old paper says that
The 24-h LD50 was established as 14 ml/kg in rats and 24 ml/kg in rabbits
so I am assuming that the first LD50 is for humans.