"Silica gel is irritating to the respiratory tract and may cause irritation of the digestive tract. Dust from the beads may cause irritation to the skin and eyes, so precautions should be taken."

Also from the same page:

"Silica gel, also referred to as silicon dioxide or synthetic amorphous silica (SAS), is listed by the FDA in the United States as generally recognized as safe (GRAS), meaning it can be added to food products without needing approval. Silica is allowed to be added to food in the US at up to 2% as permitted under 21 CFR 172.480. In the EU, it can be in up to 5% concentrations. In 2018, a re-evaluation by the EFSA Panel on Food Additives and Nutrient Sources added to Food found no indications of toxicity even at the highest estimates of exposure level."

"Listed uses include: anticaking agent, defoaming agent, stabilizer, adsorbent, carrier, conditioning agent, chill proofing agent, filter aid, emulsifying agent, viscosity control agent, and anti-settling agent. Silica can be found commonly in foods including baked goods, spices and herbs, dairy products, cocoa products, and more."

Irritation is not toxicity? I guess not, but it seems like irritation should be also be considered when adding it to food. Long term irritation can be a health issue.

They are probably quite harmful if large pieces get eaten in non-small quantities. Or powder in large quantities that are not previously mixed with a liquid. But I don't know of anybody that tested that.