Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
delecti
3 days ago
|
parent
|
context
|
favorite
| on:
Facebook's Fascination with My Robots.txt
I'm sure their crawler can handle a zip bomb. Plus it might interpret that as "this site doesn't have a robots.txt" and start scraping that OP is trying to prevent with their current robots.txt.
help
marginalia_nu
3 days ago
|
next
[–]
Pretty sure every crawler can. You kinda have to go out of your way not to, given how the gzread API looks.
https://refspecs.linuxbase.org/LSB_3.0.0/LSB-Core-generic/LS...
reply
1e1a
3 days ago
|
prev
[–]
Could allow only the path to the zip bomb for this user agent.
reply
FartyMcFarter
3 days ago
|
parent
[–]
That will work once at most and then quickly get fixed.
reply
xp84
3 days ago
|
root
|
parent
|
next
[–]
Yeah it seems like this team takes a really tough stance on obvious bugs
reply
esseph
3 days ago
|
root
|
parent
|
prev
[–]
Are you so sure? :)
reply
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
