5. Web Data, APIs and Google Sheets
5.7 Refreshing Web Data and Ethics
- Refresh: public URLs and APIs refresh in the Service without a gateway. Some HTML-scraping functions may need a gateway, and dynamic URLs need the RelativePath pattern (5.3). Test a refresh in the Service early.
- Rate limits: do not call an API or website thousands of times per refresh. Cache what you can and request only the pages you need.
- Legal and ethical note: Always read and follow a website's terms of use and its robots.txt guidance. Do not scrape sites that prohibit it, do not collect personal data without a lawful reason, and prefer official APIs or downloadable datasets. When in doubt, ask the data owner for permission.
Practice task
Write a short checklist (5 points) that you will follow before scraping any website for a college project.
Ravindra Bagale's Tip
I have seen some learners schedule a scraping report to refresh every half hour against a small website. Refresh only as often as the data really changes, respect robots.txt and the site's terms of use, and prefer an official API or downloadable file when one exists. It's very simple – just make it a habit.
Ravindra Bagale's Tip – मराठी
मी पाहिलं आहे की काही learners छोट्या website वर scraping report दर अर्ध्या तासाला refresh होईल असा schedule करतात. Data खरंच जितक्या वेळा बदलतो तितक्याच वेळा refresh करा, robots.txt आणि site च्या terms of use चा आदर करा, आणि official API किंवा downloadable file असेल तर तीच वापरा. एकदम simple आहे, फक्त सवय लावून घ्या.
Ravindra Bagale's Tip – हिंदी
मैंने देखा है कि कुछ learners किसी छोटी website पर scraping report को हर आधे घंटे refresh होने के लिए schedule कर देते हैं. उतनी ही बार refresh करो जितनी बार data सच में बदलता है, robots.txt और site की terms of use का सम्मान करो, और official API या downloadable file हो तो वही इस्तेमाल करो. बहुत आसान है, बस आदत बना लो.
Thodkyaat sangaycha tar (quick recap)
Base URL static theva, badalnare bhag RelativePath madhe taaka, fakt lagnare JSON fields expand kara, aani refresh garjepurtach theva. Site chya niyamancha adar kara. Aata pudhe jaauya Power Query chya duniyet!