<![CDATA[Basic SEO for Web Developers Part 2]]>1. Robots Exclusion Standard

  • Robots.txt
Location: https://example.com/robots.txt

User-agent: googlebot
Disallow: /example.html
Sitemap: https://example.com/sitemap.xml

##More information at http://www.robotstxt.org/robotstxt.html
  • X-Robots
Location: Sent in the HTTP headers

X-Robots-Tag: noindex

##More information at http://noarchive.net/xrobots/
  • Meta Robots
Location: In the html <head>

<meta name="ROBOT NAME" content="ARGUMENTS" />

##More information athttp://www.robotstxt.org/meta.html

  • Robots Best Practices

    1. Only Meta Robots and X-Robots remove URLs from search results
    2. Don't block CSS or JavaScript files with robots.txt

  • Arguments can be:

    1. Nofollow (do not follow links)
    2. Noindex (do not index)
    3. Noarchive (do not archive)
    4. NoODP (Do not show Open Directory Project description)...Or combined (noindex, nofollow)
    5. If the robots <META> tag is not defined, the default is "INDEX,FOLLOW"

2. Important User Agents

  • For robots.txt, robots meta tags, and X-Robots-Tag

    Googlebot (can be used as default for most Google crawlers)
Googlebot-News
Googlebot-Image
AdsBot-Google
Baiduspider
FacebookExternalHit
Slurp
Mediapartners-Google (Mobile Adsense) or Mediapartners
Googlebot-Mobile
Googlebot-Video
Bingbot
Yandexbot
Applebot
Twitterbot
Rogerbot

3. Sitemap Syntax

  • XML Sitemaps

    <?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    <url>
        <loc>https://example.com/</loc> 
        <lastmod>2015-01-01</lastmod> 
        <changefreq>monthly</changefreq> 
        <priority>0.9</priority>
    </url> 
</urlset>

  • Sitemap Index File

    <?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    <sitemap>
        <loc>https://example.com/sitemap1.xml.gz</loc>
        <lastmod>2015-01-01T18:23:17+00:00</lastmod>
    </sitemap>
    <sitemap>
        <loc>https://example.com/sitemap2.xml.gz</loc>
        <lastmod>2015-01-01</lastmod>
    </sitemap>
</sitemapindex>

  • Default Locations Can Be:

    https://example.com/sitemap.xml
https://example.com/sitemap.xml.gz
https://example.com/sitemap.gz

  • Other Common Sitemap Types:

    Mobile
News
Image
Video

4. Pagination

  • Use rel="next" andrel="prev" in the <head> sectionto indicate the relationship between paginated URLs

  • First Page - https://example.com/article

    <link rel="next" href="https://example.com/article?pg=2">

  • Second Page - https://example.com/article?pg=2

    <link rel="prev" href="https://example.com/article">
<link rel="next" href="https://example.com/article?pg=3">

  • Final Page - https://example.com/article?pg=3

    <link rel="prev" href="https://example.com/article?pg=2">

#More information at http://mz.cm/rel-next

I'll post next part asap.
Thank you all for your time in reading this!

]]>https://lankadevelopers.lk/topic/32/basic-seo-for-web-developers-part-2RSS for NodeSat, 18 Apr 2026 04:47:06 GMTSun, 09 Dec 2018 18:08:32 GMT60<![CDATA[Reply to Basic SEO for Web Developers Part 2 on Wed, 12 Dec 2018 05:47:53 GMT]]>@fern & @rachitha ur welcome :smiley:

]]>https://lankadevelopers.lk/post/176https://lankadevelopers.lk/post/176Wed, 12 Dec 2018 05:47:53 GMT<![CDATA[Reply to Basic SEO for Web Developers Part 2 on Wed, 12 Dec 2018 05:46:19 GMT]]>Thank you...

]]>https://lankadevelopers.lk/post/175https://lankadevelopers.lk/post/175Wed, 12 Dec 2018 05:46:19 GMT<![CDATA[Reply to Basic SEO for Web Developers Part 2 on Mon, 10 Dec 2018 17:34:56 GMT]]>Nice post... 😍

]]>https://lankadevelopers.lk/post/172https://lankadevelopers.lk/post/172Mon, 10 Dec 2018 17:34:56 GMT<![CDATA[Reply to Basic SEO for Web Developers Part 2 on Mon, 10 Dec 2018 04:37:48 GMT]]>@root ur welcome, sure i'll link

]]>https://lankadevelopers.lk/post/171https://lankadevelopers.lk/post/171Mon, 10 Dec 2018 04:37:48 GMT<![CDATA[Reply to Basic SEO for Web Developers Part 2 on Mon, 10 Dec 2018 02:10:24 GMT]]>Thnx bro. can you link this part with previous part.

]]>https://lankadevelopers.lk/post/169https://lankadevelopers.lk/post/169Mon, 10 Dec 2018 02:10:24 GMT