Basic SEO for Web Developers Part 2
-
1. Robots Exclusion Standard
- Robots.txt
Location: https://example.com/robots.txt User-agent: googlebot Disallow: /example.html Sitemap: https://example.com/sitemap.xml ##More information at http://www.robotstxt.org/robotstxt.html- X-Robots
Location: Sent in the HTTP headers X-Robots-Tag: noindex ##More information at http://noarchive.net/xrobots/- Meta Robots
Location: In the html <head> <meta name="ROBOT NAME" content="ARGUMENTS" /> ##More information athttp://www.robotstxt.org/meta.html-
Robots Best Practices
- Only Meta Robots and X-Robots remove URLs from search results
- Don't block CSS or JavaScript files with robots.txt
-
Arguments can be:
- Nofollow (do not follow links)
- Noindex (do not index)
- Noarchive (do not archive)
- NoODP (Do not show Open Directory Project description)...Or combined (noindex, nofollow)
- If the robots <META> tag is not defined, the default is "INDEX,FOLLOW"
2. Important User Agents
-
For robots.txt, robots meta tags, and X-Robots-Tag
Googlebot (can be used as default for most Google crawlers) Googlebot-News Googlebot-Image AdsBot-Google Baiduspider FacebookExternalHit Slurp Mediapartners-Google (Mobile Adsense) or Mediapartners Googlebot-Mobile Googlebot-Video Bingbot Yandexbot Applebot Twitterbot Rogerbot
3. Sitemap Syntax
-
XML Sitemaps
<?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url> <loc>https://example.com/</loc> <lastmod>2015-01-01</lastmod> <changefreq>monthly</changefreq> <priority>0.9</priority> </url> </urlset> -
Sitemap Index File
<?xml version="1.0" encoding="UTF-8"?> <sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <sitemap> <loc>https://example.com/sitemap1.xml.gz</loc> <lastmod>2015-01-01T18:23:17+00:00</lastmod> </sitemap> <sitemap> <loc>https://example.com/sitemap2.xml.gz</loc> <lastmod>2015-01-01</lastmod> </sitemap> </sitemapindex> -
Default Locations Can Be:
https://example.com/sitemap.xml https://example.com/sitemap.xml.gz https://example.com/sitemap.gz -
Other Common Sitemap Types:
Mobile News Image Video
4. Pagination
-
Use rel="next" andrel="prev" in the <head> sectionto indicate the relationship between paginated URLs
-
First Page - https://example.com/article
<link rel="next" href="https://example.com/article?pg=2"> -
Second Page - https://example.com/article?pg=2
<link rel="prev" href="https://example.com/article"> <link rel="next" href="https://example.com/article?pg=3"> -
Final Page - https://example.com/article?pg=3
<link rel="prev" href="https://example.com/article?pg=2"> #More information at http://mz.cm/rel-next
I'll post next part asap.
Thank you all for your time in reading this! -
Thnx bro. can you link this part with previous part.
-
@root ur welcome, sure i'll link
-
Nice post... 😍
-
Thank you...
-
